MongoDB .local Houston 2019: MongoDB Atlas Data Lake Technical Deep Dive
•
1 like•254 views
MongoDB Atlas Data Lake is a new service offered by MongoDB Atlas. Many organizations store long term, archival data in cost-effective storage like S3, GCP, and Azure Blobs. However, many of them do not have robust systems or tools to effectively utilize large amounts of data to inform decision making. MongoDB Atlas Data Lake is a service allowing organizations to analyze their long-term data to discover a wealth of information about their business. This session will take a deep dive into the features that are currently available in MongoDB Atlas Data Lake and how they are implemented.
Report
Share
MongoDB .local Houston 2019: MongoDB Atlas Data Lake Technical Deep Dive
- 2. Atlas Data Lake Technical Deep-Dive Subhead for the presentation goes here Craig Wilson, Senior Staff Engineer, MongoDB
- 3. State of Affairs Businesses have a humongous amount of data • IDC predicts that by 2025 global data will reach 175 Zettabytes and 49% of it will reside in the public cloud. Cloud storage is cost-effective Cloud storage is hard to operationalize
- 4. A New Service Offered by MongoDB Atlas Access long-term data Query long-term data Analyze long-term data
- 5. Requirements Look and act like MongoDB Access customer’s data securely Handle queries over vast amounts of data Handle long-running queries Efficient use of resources
- 7. Language Must be able to communicate with our drivers Written in Go Implemented a TCP server Used mongo-go-driver’s wireprotocol package Used mongo-go-driver's bson package
- 8. Security Must have the same security as MongoDB Users configured in Atlas Implemented MongoDB’s security model Require the use of TLS + SNI(Server Name Indicator)
- 9. Behavior Must behave like MongoDB Implemented commands for a read-only server Used the server’s aggregation engine
- 10. Customer’s Data
- 11. Security: Customers Customers have complete control Provide us with an IAM Role Configure your buckets Configure your users in Atlas
- 12. Security: Atlas Atlas controls access to your data Storage of IAM Role Temporary Credentials
- 13. Configuration Customers control their data layout Stores Databases, Collections DataSources CollectionCollection Store Store DataSource DataSource DataSource
- 14. Configuration: File Formats • BSON (gzipped) • JSON (gzipped) • Avro (gzipped) • CSV/TSV (gzipped) • Parquet • XLSX
- 15. Configuration (S3 Bucket): ent-archive /archive/customers - a-m.json - n-z.json /archive/invoices - 2019 - 1.parquet - 2.parquet - 2018 - 1.parquet - 2017.json.gz - 2016.json.gz
- 16. Configuration: Store s3 : { name: "ent-archive", bucket: "ent-archive", region: "us-east-1", prefix: "/archive/" }
- 17. Configuration: Data history: { customers: [{ store: "ent-archive", definition: "/customers/*" }], invoices: [{ store: "ent-archive", definition: "/invoices/{year int}/*" }, { store: "ent-archive", definition: "/invoices/{year int}.json.gz" }] }
- 18. Configuration: Data (Future) history: { invoices: [{ store: "ent-archive", definition: "/invoices/{year int}/*" }, { store: "ent-archive", definition : "/invoices/{year int}.json.gz }, { store: "atlas", db: "customers", collection: "invoices" }] }
- 19. Queries
- 20. Processing MQL à Distributed MQL Parse Parallelize Distribute
- 21. Architecture Atlas Control Control Plane Compute Plane Data Plane DataLake Frontend DataLake Agent Load Balancer Load Balancer DataLake Frontend DataLake Agent Load Balancer Load Balancer DataLake Frontend DataLake Agent Load Balancer Load Balancer
- 23. Query Example: $limit Map: { $match: { year: { $gt: 2000 } } } { $limit: 10 } Reduce: { $limit: 10 } { $match: { year: { $gt: 2000 } } } { $limit: 10 }
- 24. Query Example: $group Map: { $group: { _id: "$year", totalAvg_sum: { $sum: "$amount" }, totalAvg_count: { $sum: 1 } } } Reduce: { $group: { _id: "$_id", totalAvg_sum: { $sum: "$totalAvg_sum" }, totalAvg_count: { $sum: "$totalAvg_count" } } } Finalize: { $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum", "$totalAvg_count"] } } } { $group: { _id: "$year", totalAvg: { $avg: "amount" } } }
- 25. Future
- 26. Future More supported MongoDB operators. $out $merge Geo operators Full Text Search
- 31. Craig Wilson Senior Staff Engineer, MongoDB
- 32. Thank You!