Spark and MongoDB
Modern architectures are moving away from a "one size fits all" approach. We are well aware that we need to use the best tools for the job. Given the large selection of options available today, chances are that you will end up managing data in MongoDB for your operational workload and with Spark for your high speed data processing needs. Description: When we model documents or data structures there are some key aspects that need to be examined not only for functional and architectural purposes but also to take into consideration the distribution of data nodes, streaming capabilities, aggregation and queryability options and how we can integrate the different data processing software, like Spark, that can benefit from subtle but substantial model changes. A clear example is when embedding or referencing documents and their implications on high speed processing. Over the course of this talk we will detail the benefits of a good document model for the operational workload. As well as what type of transformations we should incorporate in our document model to adjust for the high speed processing capabilities of Spark. We will look into the different options that we have to connect these two different systems, how to model according to different workloads, what kind of operators we need to be aware of for top performance and what kind of design and architectures we should put in place to make sure that all of these systems work well together. Over the course of the talk we will showcase different libraries that enable the integration between spark and MongoDB, such as MongoDB Hadoop Connector, Stratio Connector and MongoDB Spark Native Connector. By the end of the talk I expect the attendees to have an understanding of: How they connect their MongoDB clusters with Spark Which use cases show a net benefit for connecting these two systems What kind of architecture design should be considered for making the most of Spark + MongoDB How documents can be modeled for better performance and operational process, while processing these data sets stored in MongoDB. The talk is suitable for: Developers that want to understand how to leverage Spark Architects that want to integrate their existing MongoDB cluster and have real time high speed processing needs Data scientists that know about Spark, are playing with Spark and want to integrate with MongoDB for their persistency layer
Spark and MongoDB
- 2. Spark in the Leaf
- 5. "BigDataSpain (2014) is great, hope I can make it next year too" Wish making conference!
- 6. "BigDataSpain (2015) is great, hope I can win la Loteria"
- 7. 7 Agenda Spark + MongoDB Connectors Use Cases Demo
- 8. By now, you should have heard about MongoDB Unless you've been living under a rock for the last few years!
- 9. 9 MongoDB GENERAL PURPOSE DOCUMENT DATABASE OPEN-SOURCE
- 10. MongoDB is Fully Featured
- 11. 11 Apache Spark
- 12. 12 Spark is Taylor Swift of Big Data
- 13. 13 Agenda Spark Taylor Swift + MongoDB Connectors Use Cases Demo
- 14. 14 Apache Spark Taylor Swift
- 15. Spark Stack Spark SQL Spark Streaming MLIB GraphX Apache Spark Seamless integration with SQL using DataFrame API. Also supports HIVE SQL Fast Feed data processing API. Designed for Fault Tolerance and bridges streaming with batch processing MLib is Spark machine learning algorithms trick bag. Spark graph library
- 16. Spark Stack Spark SQL Spark Streaming MLIB GraphX Apache Spark
- 17. Spark Stack Spark SQL Spark Streaming MLIB GraphX Apache Spark
- 18. Spark Stack Spark SQL Spark Streaming MLIB GraphX Apache Spark
- 19. Spark + MongoDB
- 20. 20 Data Management Offline Processing Analytics Data Warehousing OLTP Applications Fine grained operations
- 21. The image cannot be displayed. Your computer21 Delivering User Relevancy • Integrate data from many sources • Fast-cycle analytics • Real-time • Reliable
- 22. Fraud Detection I'm so in love!
- 23. Fraud Detection I'm so in love! Me, too<3 Now send me your CC number ? Ok, XXXX-123-zzz $$$
- 24. Fraud Detection
- 25. Workloads Chat App Login User Profile Contacts Messages … Spark Fraud Detection Segmentation Recommendations HDFS HDFS HDFS Archiving Data Crunching
- 26. 26 Wearable Devices Embedded Systems Internet of Things Embedded medical devices
- 27. The image cannot be displayed. Your computer27 Access complete patient history Avoid of conflicting prescriptions Clinical trials
- 28. High Speed Document Design
- 29. Time Series db.ticks.find() { _id: 'MSFT_12', type: 'Open', date: ISODate("2015-07-12 10:00"), volume: 1699342, minutes: { "0": 12.9, "1": 14.4, ... "59": 15.8 } } Resource Type When Series
- 31. Very High Speed
- 32. > mongod --storageEngine wiredTiger
- 33. > mongod On Upcoming 3.2
- 34. 34 MongoDB Storage Engines Content Repo IoT Sensor Backend Ad Service Customer Analytics Archive MongoDB Query Language (MQL) + Native Drivers MongoDB Document Data Model MMAP V1 WT In-Memory ? ? Supported in MongoDB 3.0 Future Possible Storage Engines Management Security Experimental
- 35. Spark Streaming
- 37. 37 Spark Streaming Twitter Feed { "statuses": [ { "coordinates": null, "favorited": false, "truncated": false, "created_at": "Mon Sep 24 03:35:21 +0000 2012", "id_str": "250075927172759552", "entities": { "urls": [ ], "hashtags": [ { "text": "freebandnames", "indices": [ 20, 34 ] } ], "user_mentions": [] } } }
- 38. 38 Spark Streaming Spark { "statuses": [ { "coordinates": null, "favorited": false, "truncated": false, "created_at": "Mon Sep 24 03:35:21 +0000 2012", "id_str": "250075927172759552", "entities": { "urls": [ ], "hashtags": [ { "text": "freebandnames", "indices": [ 20, 34 ] } ], "user_mentions": [] } } } { "time": "Mon Sep 24 03:35", "freebandnames": 1 } { "statuses": [ { "coordinates": null, "favorited": false, "truncated": false, "created_at": "Mon Sep 24 03:35:21 +0000 2012", "id_str": "250075927172759552", "entities": { "urls": [ ], "hashtags": [ { "text": "freebandnames", "indices": [ 20, 34 ] } ], "user_mentions": [] } } } { "statuses": [ { "coordinates": null, "favorited": false, "truncated": false, "created_at": "Mon Sep 24 03:35:21 +0000 2012", "id_str": "250075927172759552", "entities": { "urls": [ ], "hashtags": [ { "text": "freebandnames", "indices": [ 20, 34 ] } ], "user_mentions": [] } } } { "statuses": [ { "coordinates": null, "favorited": false, "truncated": false, "created_at": "Mon Sep 24 03:35:21 +0000 2012", "id_str": "250075927172759552", "entities": { "urls": [ ], "hashtags": [ { "text": "freebandnames", "indices": [ 20, 34 ] } ], "user_mentions": [] } } } { "time": "Mon Sep 24 03:35", "freebandnames": 4 }
- 39. 39 Capped Collection Spark Streaming { "time": "Mon Sep 24 03:35", "freebandnames": 4 } { "time": "Mon Sep 24 03:40", "bigdataspain": 400 } { "time": "Mon Sep 24 03:50", "bigdataspain": 7556 } { "time": "Mon Sep 24 03:50", "itshappending": 100 } Tailable Cursor
- 40. Spark SQL
- 41. MongoDB Hadoop Connector Spark HDFS HDFS HDFS MongoDB Hadoop Connector MongoDB Shard
- 42. MongoDB Hadoop Connector Spark HDFS HDFS HDFS MongoDB Hadoop Connector MongoDB Shard YARN
- 43. 43 MongoDB Hadoop Connector Positive Not So Good Battle Tested Not the fastest thing Integrated with existing Hadoop components Not dedicated to Spark Supports HIVE and PIG Dependent on HDFS http://docs.mongodb.org/ecosystem/tutorial/getting-started-with-hadoop/
- 45. 45 Stratio Spark-MongoDB https://github.com/Stratio/spark-mongodb Spark HDFS HDFS HDFS MongoDB Shard Stratio Spark-MongoDB
- 46. 46 Stratio Spark-MongoDB val mcInputBuilder = MongodbConfigBuilder(Map(Host -> List("localhost:27017"), Database -> "marketdata", Collection -> "minbars", SamplingRatio -> 1.0, WriteConcern -> MongodbWriteConcern.Normal)) val readConfig = mcInputBuilder.build() Database Collec9on Sampling Ra9o Write Concern
- 47. 47 Stratio Spark-MongoDB val sqlContext = new HiveContext(sc) val dfOneMin = sqlContext.fromMongoDB(readConfig)
- 48. 48 Stratio Spark-MongoDB val dfFiveMinForMonth = sqlContext.sql( """ SELECT m.Symbol, m.OpenTime as Timestamp, m.Open, m.High, m.Low, m.Close FROM ... FROM minbars) as m WHERE unix_timestamp(m.CloseTime, 'yyyy-MM-dd HH:mm') - unix_timestamp(m.OpenTime, 'yyyy-MM-dd HH:mm') = 60*4""" )
- 49. 49 Stratio Spark-MongoDB https://github.com/Stratio/spark-mongodb Spark HDFS HDFS HDFS MongoDB Shard Stratio Spark-MongoDB
- 50. 50 DC West DC West DC West Stratio Spark-MongoDB https://github.com/Stratio/spark-mongodb Spark MongoDB Shard Spark Spark
- 51. Demo
- 54. Future
- 55. 55 What to expect • We are working on a dedicated Spark Connector for MongoDB • Stratio Connector is great but: – Some Operations are actually faster if performed using Aggregation Framework • Better Integration with upcoming 3.2 Async Java Driver – Specially for the Apache Streaming Support
- 56. MongoDB Days 2015 05 November, 2015 London https://www.mongodb.com/events/mongodb-days-uk