A machine learning and data science pipeline for real companies
•Download as PPTX, PDF•
1 like•1,659 views
Comcast is one of the largest cable and telecommunications providers in the country built on decades of mergers, acquisitions, and subscriber growth. The success of our company depends on keeping our customers happy and how quickly we can pivot with changing trends and new technologies. Data abounds within our internal data centers and edge networks as well as both the private and public cloud across multiple vendors. Within such an environment and given such challenges, how do we get AI, machine learning, and data science platforms built so our company can respond to the market, predict our customers’ needs and create new revenue generating products that delight our customers? If you don’t happen to be our friends and colleagues at Google, Facebook, and Amazon, what are technologies, strategies, and toolkits you can employ to bring together disparate data sets and quickly get them into the hands of your data scientists and then into your own production systems for use by your customers and business partners? We’ll explore our journey and evolution and look at specific technologies and decisions that have gotten us to where we are today and demo how our platform works. Speaker Ray Harrison, Comcast, Enterprise Architect Prashant Khanolkar, Comcast, Principal Architect Big Data
Report
Share
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
Recommended for you
More Related Content
What's hot
What's hot (20)
Similar to A machine learning and data science pipeline for real companies
Similar to A machine learning and data science pipeline for real companies (20)
More from DataWorks Summit
More from DataWorks Summit (20)
Recently uploaded
Recently uploaded (20)
A machine learning and data science pipeline for real companies
- 1. A Data Science Pipeline for Real Companies Comcast’s Approach to Multi-datacenter, Cloud and On- premise Machine Learning
- 2. “Comcast brings together the best in media and technology. We drive innovation to create the world's best entertainment and online experiences.” High Speed Internet Video Home Automation Digital Voice Xfinity MobileContent $84b (2017) 29m Customers
- 3. “Comcast brings together the best in media and technology. We drive innovation to create the world's best entertainment and online experiences.” High Speed Internet Video Home Automation Digital Voice Xfinity MobileContent $84b (2017) 29m Customers
- 5. • Predictive network analysis • Customer premise self-healing • Comcast network self-healing • Trouble-ticket prioritization • Customer self-help (voice and text flows) • Customer Retention Use Cases: It’s All About The Customer
- 6. Our starting point! Internal Data Centers Cloud Based Infrastructure E1 E2 E3 Predictions Next Big Thing Where’s the data? How do I access it? What tools do I have? ? Where can I find information about data?
- 10. Our Challenges Security Diversity of Skills
- 11. Our Challenges Security Diversity of Skills Discoverability
- 12. FAST Provide frameworks, capabilities that allow for rapid deployment. SIMPLE & TRANSPARENT Develop capabilities to promote self-service and ease of access to data. CONSISTENT & SECURE Provide a universal security framework to govern all data under the Big Data Domain. FULLY AUTOMATED Provide a robust operational model allowing for playback, data quality, and self-healing. Guiding Principles Gather, organize, make sense of Comcast data, and make it universally accessible to empower, enable, and transform Comcast into an insight-driven organization. Product Vision
- 13. • Avoid religious wars where possible • Whatever framework makes sense for the business problem at hand • Focus on federated access to curated data • Focus on Common APIs for Ingest, Egress and Machine Learning • Focus on metadata and discoverability for: • Enterprise Data • Enterprise Features • Trained Models • Enterprise Portal • Containerized scoring endpoints that accommodate multiple frameworks and models Approach
- 14. Shameless Lyft: Uber’s Michelangelo becomes Comcast’s Da Vinci • Focus on Art AND Science (and a smattering of creativity) • Common APIs usable from multiple frameworks using Python or Scala • Metadata is Key • About data • About features • About trained models Focus on a Common Approach to Features and Models
- 15. AT LAS Ingest API Egress API (Federation Layer) Feature Store Model Store On Premise Cloud Portal <Your Favorite Framework Here> APACHE Tools such as Presto and Alluxio Scala Python Open ML API Training Deployment Container Container Container Client
- 16. Open ML API • Reads and writes features and feature metadata • Reads and writes model metadata • Integrates with a common portal searchable by any user 16
- 17. DEMO 17
- 18. The Data Science Pipeline DX Alpha
- 19. Goals • Develop a system to manage features and models running in Spark – Based on Uber’s Michelangelo • Make it easier to build and deploy data transformations and ML models • Enhance sharing of code across data science teams • Support a variety of data science toolkits
- 20. Data Science Pipeline Components • Feature Store – Standardized approach to define data transformations – A feature is a single attribute or column in a data frame – A feature table is a set of features combined with meta data – The transformation definition is separated from the ”context” in which it is applied
- 21. Data Science Pipeline Components • Model store – Standardized approach for defining models – A model is defined by train, predict, and evaluate functions, a hyperparameter set, and associated meta data – The definition of the train, predict, and evaluate functions are separated from their application – A model may be associated with one or more trained instances, prediction data frames, and evaluation metrics
- 22. Data Science Pipeline Components • Job Scheduler/Runner – Handle streaming, scheduled, and one-time jobs – Support interdependencies between jobs
- 23. Data Science Pipeline Components • File system – Store executable objects such as jar files and notebooks – Store data frames – Store trained models and other runtime artifacts
- 24. Development Approach • Build on top of Databricks and Spark • Start with a “thin slice” proof of concept – Demonstrate basic end to end run from data exploration to model evaluation • Iterate to improve usability and tooling
- 25. What do we need to know about Feature Tables? – Descriptive Information • What data transformation does it perform? • Who’s owns this feature table? • Description of Input/output – Build/deployment information • Where’s the code? What’s the current version? • What artifacts have been deployed to the production environment? – Run information • What jobs are running or have been run? • What’s the status of these jobs? • What data sets or streams are being produced and how do I access them? • Are there performance metrics or summary statistics available?
- 26. What do we need to know about Models? • Descriptive Information – What does it do? Classification? Regression? – Who’s owns this model? – Is it supervised or unsupervised? What type of labels are required? – What features does it use? • Build/deployment information – Where’s the code? What’s the current version? – What artifacts have been deployed to the production environment? • Run information – What training / prediction / evaluation jobs are running or have run? – What’s the status of these jobs? – What data sets or streams are being produced and how do I access them? – How well is the model performing? What criteria are being used to assess this?
- 27. What actions do we need to perform? • Data exploration and development • Packaging, versioning, and deployment of ML code • Job scheduling and monitoring • Storage/discovery/retrieval of job results – Data frames – Metrics – Trained models • Discovery of and interaction with Features and Models
- 28. What Technologies Already do this? • Data exploration and development – Databricks notebooks, local IDE • Packaging, versioning, and deployment of code and metadata – Github / Jenkins / Mortar – Document store for metadata (MongoDB, Cassandra, etc.) • Job scheduling and monitoring – Airflow, Databricks Jobs API • Storage of job results – DBFS / S3, need to define standard file structure • Discovery of and interaction with Features and Models – Finding – Elastic Search or existing Thin Slice API – Reading/processing data frame artifacts – Spark, Databricks notebooks – Retrieving/viewing performance metrics - ??? – Monitoring model performance over time - ??? – Algebraic composition features and models - ???
- 29. Open questions • How do we abstract file system details and other constants? • How do we standardize ETL from other systems within Comcast? • How do we support human-labeling of data sets? • What other tools (H20, R, etc.) do we need to support? • Are there other ways we need to interact with features and models? • How do we integrate AutoML? • Other technologies that may be useful? Databricks Delta? Amazon Sagemaker?
- 30. Architecture V2: PIpelines • Pipeline Segments (same as Spark ML) – Transformers – Estimators • Pipeline: linear sequence of Pipeline Segments (same as Spark ML) – Transformation pipelines contain only Transformers – Estimation pipelines contain one or more Transformers and end with an Estimator • T E D T T T T TD D T Transformation Pipeline Estimation Pipeline
- 31. Architecture V2: PIpelines • A pipeline is just a function • It does not produce anything until supplied a specific DataFrame as input
- 32. Architecture V2: Workflows • A Workflow is a directed (acyclic?) graph of Pipelines – DataSources load data (from disk, streams, etc.) into a DataFrame – Connectors merge the output of multiple DataSources into a single DataFrame – Pipelines process the DataFrames – DataSinks receive the output of the last pipeline • Workflow rules – Workflows must end in a single Pipeline node – An Estimator Pipeline may only appear as the last node in a Workflow
- 33. Architecture V2: Workflows PT PE D D D C D C T PT D D C D C PTTransformation Workflow Training Workflow
- 34. Architecture V2: Data Sources • Potential sources of data – HTTP request – Persistent store (avro, parquet, EDW, …) – Kafka topic – Others? • Connectors could handle complex logic such as combining HTTP data with other sources before feeding into a Workflow
- 35. Architecture V2 • Components – Pipeline Segment Store • Code catalog of available transforms, estimators, and pipelines • Searchable by description, tags, and maybe by schema? – “Find me a feature of type x that is tagged y” – Workflow Store • Stores DAGs (maybe Neo4j or other graph DB?) • Integrates with Databricks to run DAGs as jobs • Periodic graph analysis to optimize Workflows – Data Source Store? • Separate system or subset of Workflow Store?
Editor's Notes
- We are a technology, entertainment and media company focused on delivering the best customer experience.