ML on Big Data: Real-Time Analysis on Time Series

Editor's Notes

1. In this section I am going go give brief introduction of architecture and business use case of our system. Our goal is to make sense of any given sensor data may it be a pressure sensor in a valve or a camera on a self-driving car. By which we may be able to take smart decisions or make predictions about the future.
2. Unstructured data doesn't have relations between columns.
3. Whatever may be the data source, we want to make a generalized solution which can handle any type of variation and enable the user to get a specialized system for this own use case
4. Assume there is oil rig with 10 machine and 100 sensors each. Say, we know that a component in a machine needs maintainance every 3 months, but in many real life situations the component may breakdown pre maturely, which may cause the company millions in down time. Having a person monitor all the sensor outputs and determine whether any component needs maintainance is not a viable solution. Our system is built to handle this use case.
5. Please have a look this data from a pressure sensor in a valve. Say as an user we know that the first anomaly is caused by miss orientation of the spring and second is caused when seal of the valve is broken. Can you suggest any methods to isolate these 2 phenomenon. Most of the traditional approaches wouldn't take in to consideration if a new type pattern emerges. next slide our solution.
6. loss of similarity
7. Knowledge base is inferences drawn from the given data.
8. This is the pipeline all the phases of our application go through. First you provide a data source, currently you can upload a local file in your computer or select it from your google chrome, formats supported are CSV and TSV. Then the data is ingested using a schema provided. You can type cast variable, join columns from multiple files, etc. Using this ingested data as our time series we can compute PCSC,
9. Simple example for say y=sin(x) time series model Prediction on time series data is one of the use case for real time time series data analytics talk 1 explains about how did we train the system and teach it the make decisions. We use the trained system for real time analytics We need some streaming or live computation framework
10. Spark streaming is a micro batch processing architecture Collects stream data into small batches and processes it Job Creation and scheduling overhead is in order of milliseconds Batch interval can be as small as 1 Second
11. We can’t rely on original data source as it cannot provide recent data once lost
12. Apache kafka is a Distributed streaming platform Publish and subscribe to streams of records. Store streams of records in a fault-tolerant way
13. Streams of time series data will be pumping data into kafka Spark will connect to kafka brokers and consume data Processes data and store to database UI server will pull data for visualization Spark is the computation layer while kafka acts as data source for streaming data We now have a streaming end to end application :  a data source to stream, a compute framework and a storage system We can connect any real time streaming source One such demo: simple aws iot with moble sensors 
14. Every batch is writing a lot of small files into storage system (HDFS) We use parquet as it is one of the best compressed fomat of data available Spark parquet format small files writing is adding up to extra overhead Reading several small files from Storage to serve UI requests is also adding up to delay
15. Sharing data between batches
16. State maintain 
17. Make one line Basic problem with live streaming is data will be broken into batches. Our mathematical model can’t rely upon a batch, needs to wait on for next batch to see if data is overflown