Amazon DynamoDB Under the Hood: How We Built a Hyper-Scale Database (DAT321) - AWS re:Invent 2018
•
20 likes•9,787 views
Come to this session to learn how Amazon DynamoDB was built as the hyper-scale database for internet-scale applications. In January 2012, Amazon launched DynamoDB, a cloud-based NoSQL database service designed from the ground up to support extreme scale, with the security, availability, performance, and manageability needed to run mission-critical workloads. This session discloses for the first time the underpinnings of DynamoDB, and how we run a fully managed nonrelational database used by more than 100,000 customers. We cover the underlying technical aspects of how an application works with DynamoDB for authentication, metadata, storage nodes, streams, backup, and global replication.
Report
Share
Amazon DynamoDB Under the Hood: How We Built a Hyper-Scale Database (DAT321) - AWS re:Invent 2018
- 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon DynamoDB Under the Hood: How We Built a Hyper-Scale Database Jaso Sorenson Senior Principal Engineer AWS/DynamoDB D A T 3 2 1
- 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is the Goal? Learn about features Understand tools Use DynamoDB more effectively
- 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda GetItem / PutItem Auto Scaling Backup Restore Streams Global Tables
- 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. GetItem
- 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. GetItem (Step 1)
- 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Sample Policy { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "dynamodb:GetItem" ], "Resource": [ "arn:aws:dynamodb:us-west-2:1234567890:table/customer" ], "Condition": { "ForAllValues:StringEquals": { "dynamodb:LeadingKeys": [ "${www.amazon.com:user_id}" ] } } } ] }
- 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. GetItem (Step 2)
- 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. PutItem
- 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DynamoDB evolved from Dynamo
- 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Paxos
- 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. PutItem
- 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Heartbeats
- 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. GetItem
- 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. CustID Customer Information 145783 { name:“Bob”, city:”London”, …} 236294 { name:“Sara”, city:”Tampa”, …} 333363 { name:“Betty”, city:”Madison”, …} 445104 { name:“James”, city:”Miami”, …} 523422 { name:“Alex”, city:”London”, …} 643145 { name:“Val”, city:”Seattle”, …} 723342 { name:“Jeff”, city:”Toledo”, …} Tables
- 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hash Value CustID Customer Information 0x9531 145783 { name:“Bob”, city:”London”, …} 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0xF355 523422 { name:“Alex”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} Hashing
- 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hash Value CustID Customer Information 0x9531 145783 { name:“Bob”, city:”London”, …} 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0xF355 523422 { name:“Alex”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …} Partitioning
- 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …}
- 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …} PutItem
- 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Eventual Consistency GetItem
- 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Storage Nodes
- 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Storage Nodes
- 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. System Management
- 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Partition Repair
- 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Partition Repair
- 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hash Value Name Projected Attributes 0x7232 Bob { name:“Bob”, …} 0x45B3 Sara { name:“Sara”, …} 0x99A5 Betty { name:“Betty”, …} 0x14C7 James { name:“James”, …} 0xA3B5 Alex { name:“Alex”, …} 0x04CA Val { name:“Val”, …} 0xDA8A Jeff { name:“Jeff”, …} 0x04CA Val { name:“Val”, …} 0x14C7 James { name:“James”, …} 0x45B3 Sara { name:“Sara”, …} 0x7232 Bob { name:“Bob”, …} 0x99A5 Betty { name:“Betty”, …} 0xA3B5 Alex { name:“Alex”, …} 0xDA8A Jeff { name:“Jeff”, …} Secondary Index
- 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Secondary Index IndexBase Table
- 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Secondary Index Old Index New Index Base Table
- 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Secondary Indices Base Table
- 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Auto Admin Partition Repair Create Tables Table Provisioning Split Partition … (DBA for DynamoDB)
- 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Table Capacity
- 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Table Capacity
- 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Trade-off
- 39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Example
- 40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …} Provisioning
- 41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Token Bucket Algorithm
- 42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Token Bucket Capacity
- 43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Full Token Bucket
- 44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Partition Bursting
- 45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Unbalanced Load 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …}
- 46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Unbalanced Load 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …} 50 throttles per second
- 47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Adaptive Capacity
- 48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Adaptive Capacity
- 49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Adaptive Capacity Active 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …}
- 50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. PID Controller
- 51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. PID Controller
- 52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. PID Controller
- 53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Too Much Allowed 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …} 150 RCU over provisioned
- 54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Partition Bursting
- 55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Throttling 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …} 50 Throttled 50 Throttled 50 Throttled
- 56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Auto Scaling
- 57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Auto Scaling
- 58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Auto Scaling 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …}
- 59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Auto Scaling
- 60. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Auto Scaling 0x12A8 236294 { name:“Sara”, city:”Tampa”, …} 0x3391 445104 { name:“James”, city:”Miami”, …} 0x6134 333363 { name:“Betty”, city:”Madison”, …} 0x9531 145783 { name:“Bob”, city:”London”, …} 0xB082 643145 { name:“Val”, city:”Seattle”, …} 0xEA8A 723342 { name:“Jeff”, city:”Toledo”, …} 0xF355 523422 { name:“Alex”, city:”London”, …}
- 61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Actual Auto-scaled Table
- 62. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Recap
- 63. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Recap
- 64. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Recap
- 65. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Recap
- 66. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Recap
- 67. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Recap
- 68. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Recap
- 69. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Provisioning Recap
- 70. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 71. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Backup and Restore
- 72. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Q: Where to durably store backups?
- 73. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Q: Where to durably store backups? A: Amazon Simple Storage Service (Amazon S3)
- 74. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Replication Log to Amazon S3
- 75. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Replication Log to Amazon S3
- 76. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Snapshot" the B-tree to Amazon S3
- 77. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Snapshot" the B-tree to Amazon S3
- 78. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Restore to a Point in Time
- 79. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Restore to a Point in Time
- 80. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Restore to a Point in Time
- 81. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Restore to a Point in Time
- 82. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Partition Splits
- 83. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Restore to a Point in Time
- 84. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. On-demand Backups
- 85. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. On-demand Backups
- 86. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Write Logs to Amazon S3
- 87. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. On-demand backup
- 88. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Without PITR enabled
- 89. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Without PITR enabled
- 90. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Without PITR enabled
- 91. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Without PITR enabled
- 92. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. With PITR enabled
- 93. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. With PITR enabled
- 94. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 95. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DynamoDB Streams
- 96. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. DynamoDB Streams
- 97. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Reading from the stream
- 98. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Writing to DynamoDB Streams
- 99. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
- 100. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Global Tables
- 101. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. High Level Architecture Region Region
- 102. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi-master replication Region Region
- 103. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi-region replication Region Region Region
- 104. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi-region Region Region Region
- 105. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. “Real” Architecture
- 106. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Initial PutItem
- 107. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. After RepOut
- 108. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Conflict Resolution • aws:rep:deleting • aws:rep:updatetime • aws:rep:updateregion
- 109. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Areas Covered GetItem / PutItem Auto Scaling Backup Restore Streams Global Tables
- 110. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Areas Not Covered
- 111. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. James Sorenson a.k.a. Jaso jaso@amazon.com
- 112. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.