SlideShare a Scribd company logo

Pattern Mining: Extracting Value from Log Data

•
•
Turi, Inc.
Turi, Inc.

Pattern mining is an unsupervised machine learning technique used to discover frequent patterns and relationships in log data. It involves finding the top frequent sets of items that occur together in the data at least a minimum number of times. There are two main approaches - candidate generation which generates and filters candidate patterns in multiple passes over the data, and pattern growth which constructs conditional databases to avoid multiple full scans. Pattern mining can be used to find commonly purchased itemsets, extract features from log data, and derive rules for recommendations.

1 of 73
Download to read offline
Pattern Mining: Getting the most out of your log data. Krishna Sridhar Staff Data Scientist, Dato Inc. krishna_srd
• Background - Machine Learning (ML) Research. - Ph.D Numerical Optimization @Wisconsin • Now - Build ML tools for data-scientists & developers @Dato. - Help deploy ML algorithms. @krishna_srd, @DatoInc About Me!
45+$and$growing$fast! About Us!
+ = Questions? • (Now) We are monitoring the chat window. • (Later) Email me srikris@dato.com. Webinars
About you?
Creating a model pipeline Ingest Transform Model Deploy Unstructured Data exploration data modeling Data Science Workflow Ingest Transform Model Deploy
GraphLab(Create( Train Model Pipeline Deploy Models Serve Requests (REST API) Monitor Services Get Live Feedback Update Pipelines Prototype & Develop Model Pipelines Update Live Experiment Deploy New Pipeline Dato(Predic2ve(Services( Dato’s Products Dato(Distributed( We can help!
Log Journey Lots of data Insights Profits
Log Mining: Pattern Mining
Logs are everywhere!
Machine Learning in Logs Source: Mining Your Logs - Gaining Insight Through Visualization
Coffee shop Coffee Shops Menu
Receipts Coffee Shops Menu
Coffee Store Logs
Frequent Pattern Mining What sets of items were bought together?
Real Applications
Real Applications
Real Applications
Log Mining: Rule Mining
Can we recommend items? Rule Mining
Real Applications
Log Mining: Feature Extraction
Feature Extraction 0 1 0 0 0 0 1 1 0 1 1 0 0 1 0 0 0 0 0 0 1 1 1 0 Receipt Space Features in Menu Space ML
3 Useful Data Mining Tasks Rule MiningPattern Mining Feature Extraction
Demo
ML is not a black-box. Transparency Learning is also about understanding. Interpretability Whatever can go wrong, will go wrong. Diagnosis Moving on
Pattern Mining Explained
Formulating Pattern Mining N distinct items → 2N itemsets
Formulating Pattern Mining Find the top K most frequent sets of length at least L that occur at least M times.
Formulating Pattern Mining Find the top K most frequent sets of length at least L that occur at least M times. - max_patterns - min_length - min_support
Pattern Mining N distinct items → 2N itemsets
Pattern Mining: Principles
Principle 1: What is frequent? A pattern is frequent if it occurs at least M times. {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D} {C, D}: 5 is frequent M = 4 {A, D}: 5 is not frequent
Principle 1: What is frequent? A pattern is frequent if it occurs at least M times. {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D} {C, D}: 5 is frequent M = 4 {A, D}: 5 is not frequent min_support
Principle 2: Apriori principle A pattern is frequent only if a subset is frequent {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D} {B, C, D} : 5 is frequent therefore {C, D} : 5 is frequent {A} : 3 is not frequent therefore {A, D} : 3 is not frequent M = 4
Two Main Algorithms • Candidate Generation - Apriori - Eclat • Pattern Growth - FP-Growth - TopK FP-Growth [GLC 1.6]
Lots of Generalizations Source: http://www.philippe-fournier-viger.com/spmf/
Candidate Generation Two phases 1. Candidate generation. 2. Candidate filtering. Exploit Apriori Principle!
Candidate Generation {AB} : ? {AC} : ? {AD} : ? {BC} : ? {BD} : ? {CD} : ? {A} : ? {B} : ? {C} : ? {D} : ? { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
Candidate Generation {AB} : ? {AC} : ? {AD} : ? {BC} : ? {BD} : ? {CD} : ? {A} : ? {B} : ? {C} : ? {D} : ? { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
Candidate Generation {AB} : ? {AC} : ? {AD} : ? {BC} : ? {BD} : ? {CD} : ? {A} : 3 {B} : 4 {C} : 5 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
Candidate Generation {AB} : ? {AC} : ? {AD} : ? {BC} : ? {BD} : ? {CD} : ? {A} : 3 {B} : 4 {C} : 5 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
Candidate Generation {AB} : ? {AC} : ? {AD} : ? {BC} : ? {BD} : ? {CD} : ? {A} : 3 {B} : 4 {C} : 5 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
Candidate Generation {AB} : ? {AC} : ? {AD} : ? {BC} : 4 {BD} : 4 {CD} : 5 {A} : 3 {B} : 4 {C} : 5 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
Candidate Generation {AB} : ? {AC} : ? {AD} : ? {BC} : 4 {BD} : 4 {CD} : 5 {A} : 3 {B} : 4 {C} : 5 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
Pattern Growth Two phases 1. Candidate filtering 2. Conditional database constructions. Avoid full scans over the data & large candidate sets!
Pattern Growth - Depth First {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {AB} : 1 {AC} : 2 {AD} : 3 {BD} : 4 {CD} : 4 {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : 0 {ABD} : 1 {ACD} : 2 {BCD} : 2 {BC} : 2
Pattern Growth - Preprocessing {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6
Pattern Growth - Depth First {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {AB} : ? {AC} : ? {AD} : ? {BD} : ? {CD} : ? {A} : ? {B} : ? {C} : ? {D} : ? { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {BC} : ?
Pattern Growth - Depth First {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {AB} : ? {AC} : ? {AD} : ? {BD} : ? {CD} : ? {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {BC} : ?
Pattern Growth - Depth First {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {AB} : ? {AC} : ? {AD} : ? {BD} : ? {CD} : ? {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {BC} : ?
Pattern Growth - Depth First {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {AB} : X {AC} : ? {AD} : ? {BD} : 4 {CD} : ? {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {BC} : 2
Pattern Growth {B} : 4 { } : 6 Call: Growth(db = DB{}, item = B, freq = {B,C,D}) DB{} {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D}
Pattern Growth {B} : 4 { } : 6 Conditional Database Construction DB{} DB{B} {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {C, D} {D} {C, D} {D}
Pattern Growth {B} : 4 { } : 6 Candidate Filtering DB{B} {C, D} {D} {C, D} {D} {D} : 4 {C} : 2 DB{} {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} DB{B} Add {BD} as frequent
Pattern Growth - Depth First {C, D} {D} {C, D} {D} {AB} : X {AC} : ? {AD} : ? {BD} : 4 {CD} : ? {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {BC} : 2
Pattern Growth Recurse: Growth(db = DB{B}, item = D, freq = {D}) DB{B} {C, D} {D} {C, D} {D} {B} : 4 { } : 6 {BD} : 4 DB{BD}
Pattern Growth - Depth First {AB} : X {AC} : ? {AD} : ? {BD} : 4 {CD} : ? {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : ? {ABD} : X {ACD} : ? {BCD} : X {BC} : 2
Compare & Constrast • Candidate Generation + Better than brute force + Filters candidate sets - Multiple passes over the data • Pattern Growth + Fewer passes over the data + Space efficient.
Compare & Constrast • Candidate Generation + Better than brute force + Filters candidate sets - Multiple passes over the data • Pattern Growth + Fewer passes over the data + Space efficient. Better choice
FP-Tree Compression Figures From Florian Verhein’s Slides on FP-Growth
FP-Growth Algorithm Figures From Florian Verhein’s Slides on FP-Growth Two phases 1. Candidate filtering. 2. Conditional database constructions.
TopK FP-Growth Algorithm Similar to FP-Growth 1. Dynamically raise min_support. 2. Estimates of min_support greatly help.
Performance on Website Logs • 1.5m events • 84k sessions • 3k unique ids
Future Work
Distributed FP-Growth Partition database on item-ids. Database
Bags + Sequences × 2 Itemset: {Item} Bags: {Item: quantity} Sequences : (item)
Model built, now what?
Creating a model pipeline Ingest Transform Model Deploy Unstructured Data exploration data modeling Data Science Workflow Ingest Transform Model Deploy
Demo
Summary Log Data Mining ≠ Rocket Science • FP-Growth for finding frequent patterns. • Find rules from patterns to make predictions. • Extract features for useful ML in pattern space.
SELECT questions FROM audience WHERE difficulty == “Easy” Thanks!
Extra Slides

More Related Content

Pattern Mining: Extracting Value from Log Data