SlideShare a Scribd company logo

Introduction to Azure DocumentDB

•Download as PPTX, PDF•
•
Denny Lee
Denny Lee

Denny Lee introduced Azure DocumentDB, a fully managed NoSQL database service. DocumentDB provides elastic scaling of throughput and storage, global distribution with low latency reads and writes, and supports querying JSON documents with SQL and JavaScript. Common scenarios that benefit from DocumentDB include storing product catalogs, user profiles, sensor telemetry, and social graphs due to its ability to handle hierarchical and de-normalized data at massive scale.

Related slideshows

Big Data and NoSQL for Database and BI Pros
Big Data and NoSQL for Database and BI ProsBig Data and NoSQL for Database and BI Pros
Big Data and NoSQL for Database and BI Pros
 
Introduction to Microsoft’s Hadoop solution (HDInsight)
Introduction to Microsoft’s Hadoop solution (HDInsight)Introduction to Microsoft’s Hadoop solution (HDInsight)
Introduction to Microsoft’s Hadoop solution (HDInsight)
 
Data Modeling for NoSQL
Data Modeling for NoSQLData Modeling for NoSQL
Data Modeling for NoSQL
 
1 of 57
Introduction to Azure DocumentDB Denny Lee, Principal Program Manager, Azure DocumentDB
Denny Lee • Principal Program Manager for Azure DocumentDB • 20+ years of experience in databases, distributed systems, data sciences, and software development at Microsoft, Concur, and Databricks • Noteable Projects: • Project Isotope: Incubation team for HDInsight • Yahoo! 24TB cube: Largest SSAS cube in production @dennylee
A Brief Overview...
{ "name": "SmugMug", "permalink": "smugmug", "homepage_url": "http://www.smugmug.com", "blog_url": "http://blogs.smugmug.com/", "category_code": "photo_video", "products": [ { "name": "SmugMug", "permalink": "smugmug" } ], "offices": [ { "description": "", "address1": "67 E. Evelyn Ave", "address2": "", "zip_code": "94041", "city": "Mountain View", "state_code": "CA", "country_code": "USA", "latitude": 37.390056, "longitude": -122.067692 } ] } Perfect for these Documents schema-agnostic JSON store for hierarchical and de-normalized data at scale
Not these documents
{ "name": "SmugMug", "permalink": "smugmug", "homepage_url": "http://www.smugmug.com", "blog_url": "http://blogs.smugmug.com/", "category_code": "photo_video", "products": [ { "name": "SmugMug", "permalink": "smugmug" } ], "offices": [ { "description": "", "address1": "67 E. Evelyn Ave", "address2": "", "zip_code": "94041", "city": "Mountain View", "state_code": "CA", "country_code": "USA", "latitude": 37.390056, "longitude": -122.067692 } ] } Perfect for these Documents schema-agnostic JSON store for hierarchical and de-normalized data at scale
Elastically Scalable Throughput + Storage
Guaranteed low latency Reads <10ms @ P99 Writes <15ms @ P99
Globally Distributed
Speaks your language
DocumentDB Query Playground Demo Code: https://www.documentdb.com/sql/demo
A Primer on Scale...
The 4 Vs of Big Data Exceeds physical limits of vertical scalabilityVolume Many different formats making integration expensiveVariety Small decision window compared to data change rateVelocity Many options or variables confounding analysisVariability
The 4 Vs of Big Data Volume Variety Velocity Variability Mobile Apps Retail Learning Telematics IoT Gaming
Let’s talk about scale. Volume and Velocity
Introduction to Azure DocumentDB
Ability to Scale from Day 1 • Bursty • Unpredictable traffic Gaming + Social Experience • Lag-free • Responsive experiences Move fast without breaking things • Iterative development needs More users, more problems
• Game scores, guilds and social membership • Leaderboards by country and social • Guild management and messaging • #1 in Apple app store for free apps <10ms 99P query latency >1M game downloads ~1B requests / day The Walking Dead, results
Caches • Scores are continuously updated • Write heavy without locality RDBMS • Scale-out requires partitioning • Schema and index management Other NoSQL Stores • Longer tail on latencies • Need to specify secondary indexes for lookups The right tool for the job ?
Fully managed NoSQL database Horizontal scaling for TB and RPS High performance, write optimized Schema agnostic indexing + Azure DocumentDB The answer for low latency @ massive scale
Fact: Managing shards is really painful. Managing shards or partitions Good news: DocumentDB has done all the heavy lifting.
Elastic scale
Measuring Throughput (Request Units) Replica gets a fixed budget of request units Request Unit/sec (RU) is the normalized currency % IOPS % CPU % Memory READ GET Document Documents INSERT POST REPLACE PUT Document Operations consume request units (RUs) Query POST Documents … Min RU/sec Max RU/sec IncomingRequests Replica Quiescent Rate limit No throttling Requests get rate limited if they exceed the SLA Customers pay for reserved request units by the hour
Elastic Scale Demo Code: https://aka.ms/docdb-benchmark
Configured @10,100 RUs ~940 writes / s ~9800 RUs
Configured @250,000 RUs ~12,100 writes / s ~128,800 RUs VM @ 99% CPU
A Global Distribution Primer…
Globally Distributed Azure DocumentDB gives you the ability circumvent the speed of light! High Availability and Disaster Recovery Replicate to any Number of regions Global low latency access Dynamically configure write and read regions
… with well-defined consistency models! Consistency Level Strong Bounded Stateless Session Eventual Total Global Order Yes Yes (outside of the “staleness window”) No, partial “session” order No Consistent prefix guarantee Yes Yes Yes Yes Monotonic Reads Yes Yes (within region and across regions outside of the staleness window) Yes (for the given session) No Monotonic Writes Yes Yes Yes Yes Read your writes Yes Yes (in the write region) Yes No stronger consistency faster performance
Global Distribution Demo Code: https://aka.ms/docdb-latency-script-nodejs
Introduction to Azure DocumentDB
Introduction to Azure DocumentDB
Introduction to Azure DocumentDB
Common Scenarios
Common scenarios Retail Gaming IoT Social Product Catalog Recommendations Personalization User Store Recommendations Personalization Event Store Device Registry Telemetry Store User Behavior Telemetry Personalization
Common scenarios IoT Event Store Device Registry Telemetry Store IoT / Sensor Data Challenges: • Hardware is relatively hard to update • Different generation of devices => different schemas (variety) • Many sensors emitting telemetry => high rate of ingestion (volume + variety)
Top 5 Automotive Manufacture in the World Telematics services include: • Safety service • Diagnostic service • Remote service Ingest and query 100+ TB of semi-structure data IoT : Vehicle Telematics
IoT : Vehicle Telematics Ingress API Inbound Interface (Web API) Raw Event Store (HOT) (DocumentDB) Aggregated Event Store (Warm) (DocumentDB) Aggregated Event Store (Cold) (Blob Storage) Outbound Interface (Web API) Message Queue (Event Hubs) Stream Processor (Stream Analytics)
Common scenarios Social + AdTech Challenges: • Ingest + Analyze Third Party Data => Who dictates schema? (variety) => How do you index? • A lot of social and user data => high rate of ingestion (volume + variety) Social User Behavior Telemetry Personalization
• Startup - Advanced Marketing Intelligence Platform • Utilizes deep learning to analyze billions of relational network connections to build a social fingerprint for each user • Extracts knowledge and cultural insights by analyzing what people choose to follow Social Analytics + Ad Technology >1B Social Media Profiles >50M Tweets per Day
• Store tweets, geo-location data, and ML results in DocumentDB • Data from each social media producer has its own schema that evolves independently • Need to iterate rapidly… no time for managing VMs Social Analytics + Ad Technology >1B Social Media Profiles >50M Tweets per Day
Before moving to DocumentDB, my developers would need to come to me to confirm that our Elasticsearch deployment would support their data or if I would need to scale things to handle it. DocumentDB removed me as a bottleneck, which has been great for me and them. Stephen Hankinson, CTO, Affinio Quote
Geospatial Support including polygons Demo Want to try? Go to DocumentDB Query Playground https://www.documentdb.com/sql/demo
Polygon Query Example https://www.keene.edu/campus/maps/tool/ Polygon of coordinates -124.630000, 48.360000 -123.870000, 46.140000 -122.230000, 45.540000 -119.170000, 45.950000 -116.920000, 45.960000 -116.990000, 49.000000 -123.050000, 49.020000 -123.150000, 48.310000 -124.630000, 48.360000
Finding Volcanos with DocumentDB https://www.documentdb.com/sql/demo
Data Sciences: Apache Spark + DocumentDB
Example: Graph Structures
Example: Graph Structures
Classic Graph Scenario: Flights vertex = airports edges = flights
Data Sciences: Apache Spark + DocumentDB Demo Notebook View: https://aka.ms/docdb-spark-graph pyView: https://aka.ms/pydocdb-spark-graph Code: https://aka.ms/docdb-spark-graph-code
Graph Calculations: Degrees, PageRank What is the most important airport (most flights in / out) tripGraph.inDegrees .sort(desc("inDegree")) .limit(10))
Advantages Data Science Scenarios • Blazing Fast IoT Scenarios • Updateable columns • Push-down predicate filtering
Advantages Blazing Fast IoT Scenarios Flight information global safety alerts weather Data Science Scenarios Device Notifications Web / REST API
Advantages Updateable Columns Flight information Data Science Scenarios Device Notifications Web / REST API { tripid: “100100”, delay: -5, time: “01:00:01” } { tripid: “100100”, delay: -30, time: “01:00:01” } {delay:-30} {delay:-30} {delay:-30}
Advantages Pushdown Predicate Filtering Data Science Scenarios {city:SEA} locations headquarter exports 0 1 country Germany city Seattle country France city Paris city Moscow city Athens Belgium 0 1 {city:SEA, dst: POR, ...}, {city:SEA, dst: JFK, ...}, {city:SEA, dst: SFO, ...}, {city:SEA, dst: YVR, ...}, {city:SEA, dst: YUL, ...}, ...
References Get direct access to the engineering team -> askdocdb@microsoft.com Resources • Schema Agnostic Indexing with DocumentDB, VLDB 2015 • Consistency Levels in DocumentDB • SQL Queries with DocumentDB • Language Integrated JavaScript queries and transactions with DocumentDB • Distribute your data globally with DocumentDB
More Resources AskDocDB@microsoft Follow @DocumentDB Use #DocumentDB documentdb.com #azure-documentDB

More Related Content

Introduction to Azure DocumentDB

  • 1. Introduction to Azure DocumentDB Denny Lee, Principal Program Manager, Azure DocumentDB
  • 2. Denny Lee • Principal Program Manager for Azure DocumentDB • 20+ years of experience in databases, distributed systems, data sciences, and software development at Microsoft, Concur, and Databricks • Noteable Projects: • Project Isotope: Incubation team for HDInsight • Yahoo! 24TB cube: Largest SSAS cube in production @dennylee
  • 4. { "name": "SmugMug", "permalink": "smugmug", "homepage_url": "http://www.smugmug.com", "blog_url": "http://blogs.smugmug.com/", "category_code": "photo_video", "products": [ { "name": "SmugMug", "permalink": "smugmug" } ], "offices": [ { "description": "", "address1": "67 E. Evelyn Ave", "address2": "", "zip_code": "94041", "city": "Mountain View", "state_code": "CA", "country_code": "USA", "latitude": 37.390056, "longitude": -122.067692 } ] } Perfect for these Documents schema-agnostic JSON store for hierarchical and de-normalized data at scale
  • 6. { "name": "SmugMug", "permalink": "smugmug", "homepage_url": "http://www.smugmug.com", "blog_url": "http://blogs.smugmug.com/", "category_code": "photo_video", "products": [ { "name": "SmugMug", "permalink": "smugmug" } ], "offices": [ { "description": "", "address1": "67 E. Evelyn Ave", "address2": "", "zip_code": "94041", "city": "Mountain View", "state_code": "CA", "country_code": "USA", "latitude": 37.390056, "longitude": -122.067692 } ] } Perfect for these Documents schema-agnostic JSON store for hierarchical and de-normalized data at scale
  • 8. Guaranteed low latency Reads <10ms @ P99 Writes <15ms @ P99
  • 11. DocumentDB Query Playground Demo Code: https://www.documentdb.com/sql/demo
  • 12. A Primer on Scale...
  • 13. The 4 Vs of Big Data Exceeds physical limits of vertical scalabilityVolume Many different formats making integration expensiveVariety Small decision window compared to data change rateVelocity Many options or variables confounding analysisVariability
  • 14. The 4 Vs of Big Data Volume Variety Velocity Variability Mobile Apps Retail Learning Telematics IoT Gaming
  • 15. Let’s talk about scale. Volume and Velocity
  • 17. Ability to Scale from Day 1 • Bursty • Unpredictable traffic Gaming + Social Experience • Lag-free • Responsive experiences Move fast without breaking things • Iterative development needs More users, more problems
  • 18. • Game scores, guilds and social membership • Leaderboards by country and social • Guild management and messaging • #1 in Apple app store for free apps <10ms 99P query latency >1M game downloads ~1B requests / day The Walking Dead, results
  • 19. Caches • Scores are continuously updated • Write heavy without locality RDBMS • Scale-out requires partitioning • Schema and index management Other NoSQL Stores • Longer tail on latencies • Need to specify secondary indexes for lookups The right tool for the job ?
  • 20. Fully managed NoSQL database Horizontal scaling for TB and RPS High performance, write optimized Schema agnostic indexing + Azure DocumentDB The answer for low latency @ massive scale
  • 21. Fact: Managing shards is really painful. Managing shards or partitions Good news: DocumentDB has done all the heavy lifting.
  • 23. Measuring Throughput (Request Units) Replica gets a fixed budget of request units Request Unit/sec (RU) is the normalized currency % IOPS % CPU % Memory READ GET Document Documents INSERT POST REPLACE PUT Document Operations consume request units (RUs) Query POST Documents … Min RU/sec Max RU/sec IncomingRequests Replica Quiescent Rate limit No throttling Requests get rate limited if they exceed the SLA Customers pay for reserved request units by the hour
  • 26. Configured @250,000 RUs ~12,100 writes / s ~128,800 RUs VM @ 99% CPU
  • 28. Globally Distributed Azure DocumentDB gives you the ability circumvent the speed of light! High Availability and Disaster Recovery Replicate to any Number of regions Global low latency access Dynamically configure write and read regions
  • 29. … with well-defined consistency models! Consistency Level Strong Bounded Stateless Session Eventual Total Global Order Yes Yes (outside of the “staleness window”) No, partial “session” order No Consistent prefix guarantee Yes Yes Yes Yes Monotonic Reads Yes Yes (within region and across regions outside of the staleness window) Yes (for the given session) No Monotonic Writes Yes Yes Yes Yes Read your writes Yes Yes (in the write region) Yes No stronger consistency faster performance
  • 35. Common scenarios Retail Gaming IoT Social Product Catalog Recommendations Personalization User Store Recommendations Personalization Event Store Device Registry Telemetry Store User Behavior Telemetry Personalization
  • 36. Common scenarios IoT Event Store Device Registry Telemetry Store IoT / Sensor Data Challenges: • Hardware is relatively hard to update • Different generation of devices => different schemas (variety) • Many sensors emitting telemetry => high rate of ingestion (volume + variety)
  • 37. Top 5 Automotive Manufacture in the World Telematics services include: • Safety service • Diagnostic service • Remote service Ingest and query 100+ TB of semi-structure data IoT : Vehicle Telematics
  • 38. IoT : Vehicle Telematics Ingress API Inbound Interface (Web API) Raw Event Store (HOT) (DocumentDB) Aggregated Event Store (Warm) (DocumentDB) Aggregated Event Store (Cold) (Blob Storage) Outbound Interface (Web API) Message Queue (Event Hubs) Stream Processor (Stream Analytics)
  • 39. Common scenarios Social + AdTech Challenges: • Ingest + Analyze Third Party Data => Who dictates schema? (variety) => How do you index? • A lot of social and user data => high rate of ingestion (volume + variety) Social User Behavior Telemetry Personalization
  • 40. • Startup - Advanced Marketing Intelligence Platform • Utilizes deep learning to analyze billions of relational network connections to build a social fingerprint for each user • Extracts knowledge and cultural insights by analyzing what people choose to follow Social Analytics + Ad Technology >1B Social Media Profiles >50M Tweets per Day
  • 41. • Store tweets, geo-location data, and ML results in DocumentDB • Data from each social media producer has its own schema that evolves independently • Need to iterate rapidly… no time for managing VMs Social Analytics + Ad Technology >1B Social Media Profiles >50M Tweets per Day
  • 42. Before moving to DocumentDB, my developers would need to come to me to confirm that our Elasticsearch deployment would support their data or if I would need to scale things to handle it. DocumentDB removed me as a bottleneck, which has been great for me and them. Stephen Hankinson, CTO, Affinio Quote
  • 43. Geospatial Support including polygons Demo Want to try? Go to DocumentDB Query Playground https://www.documentdb.com/sql/demo
  • 44. Polygon Query Example https://www.keene.edu/campus/maps/tool/ Polygon of coordinates -124.630000, 48.360000 -123.870000, 46.140000 -122.230000, 45.540000 -119.170000, 45.950000 -116.920000, 45.960000 -116.990000, 49.000000 -123.050000, 49.020000 -123.150000, 48.310000 -124.630000, 48.360000
  • 45. Finding Volcanos with DocumentDB https://www.documentdb.com/sql/demo
  • 49. Classic Graph Scenario: Flights vertex = airports edges = flights
  • 50. Data Sciences: Apache Spark + DocumentDB Demo Notebook View: https://aka.ms/docdb-spark-graph pyView: https://aka.ms/pydocdb-spark-graph Code: https://aka.ms/docdb-spark-graph-code
  • 51. Graph Calculations: Degrees, PageRank What is the most important airport (most flights in / out) tripGraph.inDegrees .sort(desc("inDegree")) .limit(10))
  • 52. Advantages Data Science Scenarios • Blazing Fast IoT Scenarios • Updateable columns • Push-down predicate filtering
  • 53. Advantages Blazing Fast IoT Scenarios Flight information global safety alerts weather Data Science Scenarios Device Notifications Web / REST API
  • 54. Advantages Updateable Columns Flight information Data Science Scenarios Device Notifications Web / REST API { tripid: “100100”, delay: -5, time: “01:00:01” } { tripid: “100100”, delay: -30, time: “01:00:01” } {delay:-30} {delay:-30} {delay:-30}
  • 55. Advantages Pushdown Predicate Filtering Data Science Scenarios {city:SEA} locations headquarter exports 0 1 country Germany city Seattle country France city Paris city Moscow city Athens Belgium 0 1 {city:SEA, dst: POR, ...}, {city:SEA, dst: JFK, ...}, {city:SEA, dst: SFO, ...}, {city:SEA, dst: YVR, ...}, {city:SEA, dst: YUL, ...}, ...
  • 56. References Get direct access to the engineering team -> askdocdb@microsoft.com Resources • Schema Agnostic Indexing with DocumentDB, VLDB 2015 • Consistency Levels in DocumentDB • SQL Queries with DocumentDB • Language Integrated JavaScript queries and transactions with DocumentDB • Distribute your data globally with DocumentDB
  • 57. More Resources AskDocDB@microsoft Follow @DocumentDB Use #DocumentDB documentdb.com #azure-documentDB

Editor's Notes

  1. Well nested, multiple properties and values
  2. Not word documents
  3. Well nested, multiple properties and values
  4. Independently scale storage and throughput. Provisioned throughput guaranteed. Elastically scale throughput from 100 to 10s of millions of requests/sec Transparent server side partitioning Optionally evict old data with TTL Cheaper than hosted OSS NoSQL databases or DynamoDB Watch “Predictable performance” module
  5. Write optimized, SSD-based database engine with low latency access Synchronous and automatic indexing at sustained ingestion rates Globally distributed with reads and writes served from local region Watch “Predictable performance” module
  6. Scale across any number of Azure regions Turn-key high availability with transparent failover Multi-homing Well-defined consistency models Watch “Achieve planet scale with DocumentDB: Multi-region replication”
  7. Rich SQL, JavaScript, MongoDB Multi-modal: key-values, column family, or documents No impedance mismatch - JavaScript is the type system Write business logic entirely in JavaScript with stored procedures and triggers Integrated multi-document transactions with snapshot isolation .NET, Java, Node, Python SDKs
  8. That’s right, you were waiting for the zombies … The Walking Dead is a show about a zombie apocalypse … or ‘walkers’ as they’re referred to in the show. The Walking Dead No Man’s Land is a mobile game based on this very successful AMC series. No Man’s Land is developed by a game company by the name of Next Games. Some of you may have heard of Next Games from Scott’s keynote this morning but for those who missed the keynote, I’ll roll the video … > PLAY VIDEO #### summary of what we just saw
  9. They will help look at a customer’s social media account (e.g. Nike’s Twitter Account) and provide reporting and analytics on the customer’s followers – including segmenting followers (e.g. you have a high number of soccer moms and athletes following you) and analyzing what kind of content is trending amongst each segment of followers. If links are included in the content – they will scrape the link and provide further analysis on any content found in the link. They are using DocumentDB as their data store for storing and querying the scraped content.