Free Code Friday - Spark Streaming with HBase

Editor's Notes

1. Spark is really cool… This is NOT a silver bullet… This is another GREAT tool to have… Stop using hammers for putting in screws.
2. a significant amount of data needs to be quickly processed in near real time to gain insights. Common examples include activity stream data from a web or mobile application, time-stamped log data, transactional data, and event streams from sensor or device networks. the stream-based approach applies processing to the data stream as it is generated. This allows near real-time processing Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data. Spark Streaming is for use cases which require a significant amount of data to be quickly processed as soon as it arrives. Example real-time use cases are: Website monitoring , Network monitoring Fraud detection Web clicks Advertising Internet of Things: sensors
3. Having data in log files is not good for real time processing
4. Its becoming important to process events as they arrive
5. Spark Streaming brings Spark's api to stream processing, letting you write streaming jobs the same way you write batch jobs. Spark streaming supports data sources such as HDFS directories, TCP sockets, Twitter, message queues and distributed stream and log transfer frameworks, such as Flume, Kafka, and Amazon Kinesis. Data Streams can be processed with Spark’s core APIS, Dataframes SQL, or machine learning APIs, and can be persisted to a filesystem, HDFS, MapR-FS, MapR-DB, HBase, or any data source offering a Hadoop OutputFormat or Spark connector. DStreams can be created from various input sources, such as Flume, Kafka, or HDFS. Once built, they offer two types of operations: transformations, which yield a new DStream, and output operations, which write data to an external system.
6. A data stream is a continuous sequence of records or events. Common examples include activity stream data from a web or mobile application, time-stamped log data, transactional data, and event streams from sensor or device networks. the stream-based approach applies processing to the data stream as it is generated. This allows near real-time processing. A stream processing architecture is typically made of a following components: Data sources – source of data streams. examples are sensor network, mobile application, web client, a log from a server, or even a thing from Internet of Things. Message bus – messaging systems such as Kafka and Flume ActiveMQ, RabbitMQ, etc. Stream processing system – framework for processing data streams. NoSql store for storing processed data must be capable of fast read and writes. HBase, Cassandra, MongoDb are popular choices. End applications – dashboards and other applications which use the processed data. DStreams can be created from various input sources, such as Flume, Kafka, or HDFS. Once built, they offer two types of operations: transformations, which yield a new DStream, and output operations, which write data to an external system.
7. Spark Streaming supports various input sources, including file-based sources and network-based sources such as socket-based sources, the Twitter API stream, Akka actors, or message queues and distributed stream and log transfer frameworks, such Flume, Kafka, and Amazon Kinesis. Spark Streaming provides a set of transformations available on DStreams; these transformations are similar to those available on RDDs. These include map, flatMap, filter, join, and reduceByKey. Streaming also provides operators such as reduce and count. These operators return a DStream made up of a single element . Unlike the reduce and count operators on RDDs, these do not trigger computation on Dstreams , they are not actions, they return another DStream. Stateful transformations maintain state across batches, they use data or intermediate results from previous batches to compute the results of the current batch. They include transformations based on sliding windows and on tracking state across time. updateStateByKey() is used to track state across events for each key. For example this could be used to maintain stats (state) by a key accessLogDStream.reduceByKey(SUM_REDUCER). Actions are output operators that, when invoked, trigger computation on the DStream. They are as follows: print: This prints the first 10 elements of each batch to the console and is typically used for debugging and testing. saveAsObjectFile, saveAsTextFiles, and saveAsHadoopFiles: These functions output each batch to a Hadoop-compatible filesystem. forEachRDD: This operator allows to apply any arbitrary processing to the RDDs within each batch of a DStream.
8. Streaming data is continuous and needs to be batched to process. Spark Streaming divides the data stream into batches of X seconds called Dstreams (discretized stream) Internally, each DStream is represented as a sequence of RDDs arriving at each time step. A DStream is a sequence of mini-batches, where each mini-batch is represented as a Spark RDD . The stream is broken up into time periods equal to the batch interval Each RDD in the stream will contain the records that are received by the Spark Streaming application during a given time window called the batch interval. Spark Streaming receives data from various input sources and groups it into small batches called the batch interval. , Each input batch forms an RDD. a Dstream is a sequence of RDDs, where each RDD has one time slice of the data in the stream.
9. Spark’s primary abstraction is a distributed collection of items called a Resilient Distributed Dataset (RDD). RDDs can be created from Hadoop InputFormats (such as HDFS files) or by transforming other RDDs.
10. An RDD is simply a distributed collection of elements. You can think of the distributed collections like of like an array or list in your single machine program, except that it’s spread out across multiple nodes in the cluster. In Spark all work is expressed as either creating new RDDs, transforming existing RDDs, or calling operations on RDDs to compute a result. Under the hood, Spark automatically distributes the data contained in RDDs across your cluster and parallelizes the operations you perform on them. So, Spark gives you APIs and functions that lets you do something on the whole collection in parallel using all the nodes.
11. actions, which return a value to the driver program after running a computation on the dataset The chain of transformations from RDD1 to RDDn are logged, and can be repeated in the event of data loss or the failure of a cluster node.
12. Your Spark Application processes the Dstream RDDs using Spark transformations like map, reduce, join … which create new RDDs. Any operation applied on a DStream translates to operations on the underlying RDDs, which, in turn, applies the transformation to the elements of the RDD. Streaming data is continuous and needs to be batched to process. Spark Streaming divides the data stream into batches of X seconds called Dstreams (discretized stream), which is a sequence of RDDs. Spark Application processes the RDDs using the Spark APIs finally the processed results of the RDD operations are returned in batches.
13. Spark Streaming supports various input sources, including file-based sources and network-based sources such as socket-based sources, the Twitter API stream, Akka actors, or message queues and distributed stream and log transfer frameworks, such Flume, Kafka, and Amazon Kinesis. Spark Streaming provides a set of transformations available on DStreams; these transformations are similar to those available on RDDs. These include map, flatMap, filter, join, and reduceByKey. Streaming also provides operators such as reduce and count. These operators return a DStream made up of a single element . Unlike the reduce and count operators on RDDs, these do not trigger computation on Dstreams , they are not actions, they return another DStream. Stateful transformations maintain state across batches, they use data or intermediate results from previous batches to compute the results of the current batch. They include transformations based on sliding windows and on tracking state across time. updateStateByKey() is used to track state across events for each key. For example this could be used to maintain stats (state) by a key accessLogDStream.reduceByKey(SUM_REDUCER). Actions are output operators that, when invoked, trigger computation on the DStream. They are as follows: print: This prints the first 10 elements of each batch to the console and is typically used for debugging and testing. saveAsObjectFile, saveAsTextFiles, and saveAsHadoopFiles: These functions output each batch to a Hadoop-compatible filesystem. forEachRDD: This operator allows to apply any arbitrary processing to the RDDs within each batch of a DStream.
14. Streaming data is continuous and needs to be batched to process. Spark Streaming divides the data stream into batches of X seconds called Dstreams (discretized stream), which is a sequence of RDDs. Spark Application processes the RDDs using the Spark APIs finally the processed results of the RDD operations are returned in batches. Output operations are similar to RDD actions in that they write data to an external system, but in Spark Streaming they run periodically on each time step, producing output in batches.
15. Spark Streaming supports various input sources, including file-based sources and network-based sources such as socket-based sources, the Twitter API stream, Akka actors, or message queues and distributed stream and log transfer frameworks, such Flume, Kafka, and Amazon Kinesis. Spark Streaming provides a set of transformations available on DStreams; these transformations are similar to those available on RDDs. These include map, flatMap, filter, join, and reduceByKey. Streaming also provides operators such as reduce and count. These operators return a DStream made up of a single element . Unlike the reduce and count operators on RDDs, these do not trigger computation on Dstreams , they are not actions, they return another DStream. Stateful transformations maintain state across batches, they use data or intermediate results from previous batches to compute the results of the current batch. They include transformations based on sliding windows and on tracking state across time. updateStateByKey() is used to track state across events for each key. For example this could be used to maintain stats (state) by a key accessLogDStream.reduceByKey(SUM_REDUCER). Actions are output operators that, when invoked, trigger computation on the DStream. They are as follows: print: This prints the first 10 elements of each batch to the console and is typically used for debugging and testing. saveAsObjectFile, saveAsTextFiles, and saveAsHadoopFiles: These functions output each batch to a Hadoop-compatible filesystem. forEachRDD: This operator allows to apply any arbitrary processing to the RDDs within each batch of a DStream.
16. These are the basic steps for Spark Streaming code: Initialize a Spark StreamingContext object. Apply transformations and output operations to DStreams. Start receiving data and processing it using streamingContext.start(). Wait for the processing to be stopped using streamingContext.awaitTermination().
17. We want to store every single event as it comes in, in a Hbase. We also want to filter for and store alarms. Daily Spark processing will store aggregated summary statistics The Spark Streaming example does the following: Reads streaming data. Processes the streaming data. Writes the processed data to an HBase Table. (non Streaming) Spark code does the following: Reads HBase Table data written by the streaming code Calculates daily summary statistics Writes summary statistics to the HBase table Column Family stats
18. The Oil Pump Sensor data comes in as comma separated value (csv) files dropped in a directory. Spark Streaming will monitor the directory and process any files created in that directory. (As stated before, Spark Streaming supports different streaming data sources; for simplicity, this example will use files.) Below is an example of the csv file with some sample data: We use a Scala case class to define the Sensor schema corresponding to the sensor data csv files, and a parseSensor function to parse the comma separated values into the sensor case class.
19. The HBase Table Schema for the streaming data is as follows: Composite row key of the pump name date and time stamp Column Family data with columns corresponding to the input data fields Column Family alerts with columns corresponding to any filters for alarming values Note that the data and alert column families could be set to expire values after a certain amount of time. The Schema for the daily statistics summary rollups is as follows: Composite row key of the pump name and date Column Family stats
20. These are the basic steps for Spark Streaming code: Initialize a Spark StreamingContext object. Using this context, create a Dstream which represents streaming data from a source Apply transformations and/or output operations to DStreams. Start receiving data and processing it using streamingContext.start(). Wait for the processing to be stopped using streamingContext.awaitTermination(). We will go through these steps showing code from our use case example Spark Streaming programs are best run as standalone applications built using Maven or sbt.
21. First we create a StreamingContext, the main entry point for streaming functionality, with a 2 second batch interval. Using this context, we can create a DStream that represents streaming data from a source . In this example we use the StreamingContext textFileStream(directory) method to create an input stream that monitors a Hadoop-compatible file system for new files and processes any files created in that directory. This ingestion type supports a workflow where new files are written to a landing directory and Spark Streaming is used to detect them, ingest them, and process the data. Only use this ingestion type with files that are moved or copied into a directory. The linesDStream represents the stream of data, each record is a line of text. Internally a DStream is a sequence of RDDs, one RDD per batch interval.
22. Next we parse the lines of data into Sensor objects, with the map operation on the linesDStream. The map operation applies the Sensor.parseSensor function on the RDDs in the linesDStream, resulting in RDDs of Sensor objects. Any operation applied on a DStream translates to operations on the underlying RDDs. the map operation is applied on each RDD in the linesDStream to generate the RDDs of the sensorDStream.
23. Next we use the DStream foreachRDD method to apply processing to each RDD in this DStream. We filter the sensor objects for low psi to create an RDD of alert sensor objects
24. Here we join the alert data with pump vendor and maintenance information (the pump vendor and maintenance information was read in and cached before streaming . Each RDD is converted to a DataFrame , registered as a temporary table and then queried using SQL. rdd.toDF().registerTempTable("sensor")
25. Next we use the DStream foreachRDD method to apply processing to each RDD in this DStream. We filter the sensor objects for low psi to create alerts, then we write the sensor and alert data to HBase by converting them to Put objects, and using the PairRDDFunctions saveAsHadoopDatasetmethod, which outputs the RDD to any Hadoop-supported storage system using a Hadoop Configuration object for that storage system (see Hadoop Configuration for HBase above).
26. The sensorRDD objects are converted to put objects then written to HBase.
27. To start receiving data, we must explicitly call start() on the StreamingContext, then call awaitTermination to wait for the streaming computation to finish.
28. The last way to leverage MapReduce in Hbase is to use the Hbase database as both a source and sink in our data flow. One example of this use case is to calculate summaries across the Hbase data and then store those summaries back in the Hbase database. For summary jobs where HBase is used as a source and a sink, then writes will be coming from the Reducer step.
29. This shows the Input and Output for the TableMapper class map method and the TableReducer class reduce method for reading from and to HBase. A scan Result object and Row key are sent to the Mapper map method one row at a time, one or more Key Value pairs are output by the Map method . The Reducer reduce method receives a Key and an iterable list of corresponding values, and outputs a put object .
30. we register the DataFrame as a table. Registering it as a table allows us to use it in subsequent SQL statements.   Now we can inspect the data.
31. we register the DataFrame as a table. Registering it as a table allows us to use it in subsequent SQL statements.   Now we can inspect the data.
32. This post will help you get started using Apache Spark DataFrames with Scala on the MapR Sandbox. The new Spark DataFrames API is designed to make big data processing on tabular data easier. A Spark DataFrame is a distributed collection of data organized into named columns that provides operations to filter, group, or compute aggregates, and can be used with Spark SQL.