Flink Case Study: Amadeus
•
3 likes•7,543 views
This document discusses streaming and parallel decision trees in Flink. It motivates the need for a classifier system that can learn from streaming data and classify both the streaming training data and new streaming data. It describes the architecture of keeping the classifier model fresh as new data streams in, allowing classification during the learning process in real-time. It also outlines decision tree algorithms and their implementation using Flink streaming.
Report
Share
Related slideshows
Flink Case Study: Amadeus
- 1. StreamingStreamingStreamingStreaming&ParallelParallelParallelParallel Decision TreeDecision TreeDecision TreeDecision Treein FlinkFlinkFlinkFlink 1 2 3 4 1 2 3 4 anwar.rizal @anrizal
- 3. MotivationMotivationMotivationMotivation Need a classifier system on streaming data The data used for learning come as a stream So are the data to be classified
- 4. MotivationMotivationMotivationMotivation $90 $90 $120 $90 $90 $150 $200 $90 $75 $90 $90 $90 $90 $90 $120 $90 Sold out Sold out $75 $90 $90 $120 $90 $90 $90 $100 $90 $120 (predicted) to increase zero to two days (predicted) to increase this week (predicted) to increase next week
- 5. MotivationMotivationMotivationMotivation FRA – NYC FRA - LON FRA - MEX Need attention Revenue decrease Need attention passenger decrease Need attention revenue decrease, cost increase
- 6. MotivationMotivationMotivationMotivation Need a classifier system on streaming data The data used for learning come as a stream So are the data to be classified
- 7. MotivationMotivationMotivationMotivation The classifier is kept fresh No need for separate batch learning/evaluation The feedback is taken into account in real time, regularly The classifier can be introspected Transparent model structure (e.g. know the tree, information gain for each split point) Known expected performance (accuracy, precision, recall, AUC) Seamless support for workflow of machine learning Data preprocessing: up/down sampling, imputations, … Feature selections Model evaluation, cross validation, MUST
- 8. MotivationMotivationMotivationMotivation The classifier is immediately available The classifier can already predict during learning When learning phase is terminated, it starts another cycle of learning The classifier has a meta-learning capability The classifier has several models different parameters It is possible to learn about the learning capability of the models NICE TO HAVE
- 9. MotivationMotivationMotivationMotivation Learning Learning & Classifying End of learning New cycle of learning Cycle of Learning, Classifying during Learning, End of Learning, Classifying, New Learning
- 10. MotivationMotivationMotivationMotivation Classifying Application Stream Learner Labeled points Classifier Predicted points Unlabeled points
- 11. MotivationMotivationMotivationMotivation Decision Tree Algorithms Outline Implementation using Flink Streaming Further Discussions
- 12. Thanks! Credit to: Yiqing Yan (Eurecom) & Tianshu Yang (Telecom Bretagne), Amadeus Interns