SlideShare a Scribd company logo

Deep Dive on Amazon Aurora

•
•
Amazon Web Services
Amazon Web Services

Amazon Aurora is a MySQL-compatible relational database engine that combines the speed and availability of high-end commercial databases with the simplicity and cost-effectiveness of open source databases. Amazon Aurora is disruptive technology in the database space, bringing a new architectural model and distributed systems techniques to provide far higher performance, availability and durability than previously available using conventional monolithic database techniques. In this session, we will do a deep-dive into some of the key innovations behind Amazon Aurora, discuss best practices and configurations, and share early customer experience from the field.

Related slideshows

Deep Dive on Amazon Aurora
Deep Dive on Amazon AuroraDeep Dive on Amazon Aurora
Deep Dive on Amazon Aurora
 
[pgday.Seoul 2022] PostgreSQL with Google Cloud
[pgday.Seoul 2022] PostgreSQL with Google Cloud[pgday.Seoul 2022] PostgreSQL with Google Cloud
[pgday.Seoul 2022] PostgreSQL with Google Cloud
 
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion Records
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion RecordsScylla Summit 2022: How to Migrate a Counter Table for 68 Billion Records
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion Records
 
1 of 37
Download to read offline
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. July 13, 2016 Deep Dive on Amazon Aurora KD Singh, AWS Solution Architect Scott Ward, AWS Solution Architect
MySQL-compatible relational database Performance and availability of commercial databases Simplicity and cost-effectiveness of open-source databases What is Amazon Aurora?
Fastest growing service in AWS history Business applications Web and mobile Content management E-commerce, retail Internet of Things Search, advertising BI, analytics Games, media Aurora customer adoption
Relational databases were not designed for cloud Multiple layers of functionality all in a monolithic stack SQL Transactions Caching Logging
Not much has changed in the last 30 years Even when you scale it out, you’re still replicating the same stack SQL Transactions Caching Logging SQL Transactions Caching Logging Application SQL Transactions Caching Logging SQL Transactions Caching Logging Application SQL Transactions Caching Logging SQL Transactions Caching Logging Storage Application
Reimagining the relational database What if you were inventing the database today? You wouldn’t design it the way we did in 1970 You’d build something  that can scale out …  that is self-healing …  that leverages existing AWS services …
A service-oriented architecture applied to databases Moved the logging and storage layer into a multitenant, scale-out database-optimized storage service Integrated with other AWS services like Amazon EC2, Amazon VPC, Amazon DynamoDB, Amazon SWF, and Amazon Route 53 for control plane operations Integrated with Amazon S3 for continuous backup with 99.999999999% durability Control PlaneData Plane Amazon DynamoDB Amazon SWF Amazon Route 53 Logging + Storage SQL Transactions Caching Amazon S3 1 2 3
SQL benchmark results 4 client machines with 1,000 connections each WRITE PERFORMANCE READ PERFORMANCE Single client machine with 1,600 connections Using MySQL SysBench with Amazon Aurora R3.8XL with 32 cores and 244 GB RAM
Reproducing these results https ://d0.a wsstat ic . com /product -m ark eting/Aurora /R DS_ Auro ra_Perf orm ance_Assessm ent_Benchm ark ing_v 1-2 .pdf AMAZON AURORA R3.8XLARGE R3.8XLARGE R3.8XLARGE R3.8XLARGE R3.8XLARGE • Create an Amazon VPC (or use an existing one) • Create 4 EC2 R3.8XL client instances to run the SysBench client. All 4 should be in the same Availability Zone (AZ) • Enable enhanced networking on your clients • Tune Linux settings (see whitepaper referenced below) • Install SysBench version 0.5 • Launch a r3.8xlarge Amazon Aurora DB instance in the same VPC and AZ as your clients • Start your benchmark! 1 2 3 4 5 6 7
Performance best practices MySQL/RDBMS practices still apply  Choose the right tool for the right job (OLAP vs. OLTP vs. NoSQL)  Create appropriate indexes  Tune your SQL code, use explain plans, performance schema  Many more… Leverage high concurrency  Aurora throughput increases with number of connections  Architect your applications to leverage high concurrency in Aurora Read scaling  Aurora offers read replicas with virtually no replication lag  Leverage multiple read replicas to distribute your reads
Performance best practices Parameter tuning  No need to migrate your performance-related MySQL parameters to Aurora  Aurora parameter groups are pretuned and already optimal in most cases Performance comparison  Don’t obsess over individual metrics (CPU, IOPS, I/O throughput)  Focus on what matters, that is application performance Other best practices  Keep query cache on  Leverage Amazon CloudWatch metrics
Advanced monitoring 50+ system/OS metrics | sorted process list view | 1–60 sec. granularity alarms on specific metrics | egress to CloudWatch Logs | integration with third-party tools ALARM
Important systems and OS metrics User System Wait IRQ Idle Nice Steal CPU utilization Rx per declared ethn Tx per declared ethn Network Sleeping Running Total Stopped Blocked Zombie Processes Process ID Process name VSS Res Mem % consumed CPU % used CPU time Parent ID Process List Free Cached Buffered Total Writeback Inactive Dirty Mapped Slab Page tables Huge pages free Huge pages rsvd Huge pages surp Huge pages size Huge pages total Swap Swap free Swap committed Memory Read latency Write latency Read throughput Write throughput Read I/O/sec. Write I/O/sec. Queue depth Read queue depth Write queue depth Free local storage Device I/O Used Total Used Inodes/% Max Inodes/% File system 1 min 5 min 15 min Load Average
Important database metrics  View database level metrics from Amazon Aurora and Amazon CloudWatch console  Perform retroactive workload analysis Select throughput Select latency DML throughput DML latency Commit throughput Commit latency DDL throughput DDL latency DB connections Active connections Login failures Buffer cache hit ratio Resultset cache hit ratio Deadlocks Blocked transactions Failed SQL statements Replica lag Replica lag maximum Replica lag minimum Free local storage
Beyond benchmarks If only real-world applications saw benchmark performance POSSIBLE DISTORTIONS Real-world requests contend with each other Real-world metadata rarely fits in the data dictionary cache Real-world data rarely fits in the buffer cache Real-world production databases need to run at high availability
Scaling user connections SysBench OLTP workload 250 tables Connections Amazon Aurora Amazon RDS MySQL 30 K IOPS (single AZ) 50 40,000 10,000 500 71,000 21,000 5,000 110,000 13,000 8x UP TO FA STER
Scaling table count SysBench write-only workload 1,000 connections, default settings Tables Amazon Aurora MySQL I2.8XL local SSD MySQL I2.8XL RAM disk RDS MySQL 30 K IOPS (single AZ) 10 60,000 18,000 22,000 25,000 100 66,000 19,000 24,000 23,000 1,000 64,000 7,000 18,000 8,000 10,000 54,000 4,000 8,000 5,000 11x UP TO FA STER Number of write operations per second
Scaling dataset size SYSBENCH WRITE-ONLY DB Size Amazon Aurora RDS MySQL 30 K IOPS (single AZ) 1 GB 107,000 8,400 10 GB 107,000 2,400 100 GB 101,000 1,500 1 TB 26,000 1,200 67x U P TO FA STER DB Size Amazon Aurora RDS MySQL 30K IOPS (single AZ) 80 GB 12,582 585 800 GB 9,406 69 CLOUDHARMONY TPC-C 136x U P TO FA STER
Running with read replicas SysBench write-only workload 250 tables Updates per second Amazon Aurora RDS MySQL 30 K IOPS (single AZ) 1,000 2.62 ms 0 s 2,000 3.42 ms 1 s 5,000 3.94 ms 60 s 10,000 5.38 ms 300 s 500x UP TO LOW ER LA G
Do fewer I/Os Minimize network packets Cache prior results Offload the database engine DO LESS WORK Process asynchronously Reduce latency path Use lock-free data structures Batch operations together BE MORE EFFICIENT How do we achieve these results? DATABASES ARE ALL ABOUT I/O NETWORK-ATTACHED STORAGE IS ALL ABOUT PACKETS/SECOND HIGH-THROUGHPUT PROCESSING DOES NOT ALLOW CONTEXT SWITCHES
Aurora cluster Amazon S3 AZ 1 AZ 2 AZ 3 Aurora primary instance Cluster volume spans 3 AZs
Aurora cluster with replicas Amazon S3 AZ 1 AZ 2 AZ 3 Aurora primary instance Cluster volume spans 3 AZs Aurora Replica Aurora Replica
I/O traffic in RDS MySQL BINLOG DATA DOUBLE-WRITELOG FRM FILES T Y P E O F W R IT E MYSQL WITH STANDBY EBS mirrorEBS mirror AZ 1 AZ 2 Amazon S3 EBS Amazon Elastic Block Store (EBS) Primary Instance Standby Instance 1 2 3 4 5 Issue write to Amazon EBS—EBS issues to mirror, acknowledge when both done Stages write to standby instance using storage level replication Issues write to EBS on standby instance I/O FLOW Steps 1, 3, 5 are sequential and synchronous This amplifies both latency and jitter Many types of write operations for each user operation Have to write data blocks twice to avoid torn write operations OBSERVATIONS 780 K transactions 7,388 K I/Os per million transactions (excludes mirroring, standby) Average 7.4 I/Os per transaction PERFORMANCE 30 minute SysBench write-only workload, 100 GB dataset, RDS Single AZ, 30 K PIOPS
I/O traffic in Aurora (database) AZ 1 AZ 3 Primary Instance Amazon S3 AZ 2 Replica Instance AMAZON AURORA ASYNC 4/6 QUORUM DISTRIBUTED WRITES BINLOG DATA DOUBLE-WRITELOG FRM FILES T Y P E O F W R IT E S 30 minute SysBench writeonly workload, 100GB dataset I/O FLOW Only write redo log records; all steps asynchronous No data block writes (checkpoint, cache replacement) 6x more log writes, but 9x less network traffic Tolerant of network and storage outlier latency OBSERVATIONS 27,378 K transactions 35x MORE 950K I/Os per 1M transactions (6x amplification) 7.7x LESS PERFORMANCE Boxcar redo log records—fully ordered by LSN Shuffle to appropriate segments—partially ordered Boxcar to storage nodes and issue write operations
I/O traffic in Aurora (storage node) LOG RECORDS Primary Instance INCOMING QUEUE STORAGE NODE S3 BACKUP 1 2 3 4 5 6 7 8 UPDATE QUEUE ACK HOT LOG DATA BLOCKS POINT IN TIME SNAPSHOT GC SCRUB COALESCE SORT GROUP PEER TO PEER GOSSIPPeer Storage Nodes All steps are asynchronous Only steps 1 and 2 are in the foreground latency path Input queue is 46x less than MySQL (unamplified, per node) Favors latency-sensitive operations Use disk space to buffer against spikes in activity OBSERVATIONS I/O FLOW ① Receive record and add to in-memory queue ② Persist record and acknowledge ③ Organize records and identify gaps in log ④ Gossip with peers to fill in holes ⑤ Coalesce log records into new data block versions ⑥ Periodically stage log and new block versions to S3 ⑦ Periodically garbage-collect old versions ⑧ Periodically validate CRC codes on blocks
Asynchronous group commits Read Write Commit Read Read T1 Commit (T1) Commit (T2) Commit (T3) LSN 10 LSN 12 LSN 22 LSN 50 LSN 30 LSN 34 LSN 41 LSN 47 LSN 20 LSN 49 Commit (T4) Commit (T5) Commit (T6) Commit (T7) Commit (T8) LSN GROWTH Durable LSN at head node COMMIT QUEUE Pending commits in LSN order TIME GROUP COMMIT TRANSACTIONS Read Write Commit Read Read T1 Read Write Commit Read Read Tn TRADITIONAL APPROACH AMAZON AURORA Maintain a buffer of log records to write out to disk Issue write operations when buffer is full, or time out waiting for write operations First writer has latency penalty when write rate is low Request I/O with first write, fill buffer till write picked up Individual write durable when 4 of 6 storage nodes acknowledge Advance DB durable point up to earliest pending acknowledgement
Re-entrant connections multiplexed to active threads Kernel-space epoll() inserts into latch-free event queue Dynamically size threads pool Gracefully handles 5000+ concurrent client sessions on r3.8xl Standard MySQL—one thread per connection Doesn’t scale with connection count MySQL EE—connections assigned to thread group Requires careful stall threshold tuning CLIENTCONNECTION CLIENTCONNECTION LATCH FREE TASK QUEUE epoll() MYSQL THREAD MODEL AURORA THREAD MODEL Adaptive thread pool
I/O traffic in Aurora (read replica) PAGE CACHE UPDATE Aurora Master 30% Read 70% Write Aurora Replica 100% New Reads Shared Multi-AZ Storage MySQL Master 30% Read 70% Write MySQL Replica 30% New Reads 70% Write SINGLE-THREADED BINLOG APPLY Data Volume Data Volume Logical: Ship SQL statements to replica Write workload similar on both instances Independent storage Can result in data drift between master and replica Physical: Ship redo from master to replica Replica shares storage; no writes performed Cached pages have redo applied Advance read view when all commits seen MYSQL READ SCALING AMAZON AURORA READ SCALING
Availability “Performance only matters if your database is up”
Storage node availability Quorum system for read/write; latency tolerant Peer-to-peer gossip replication to fill in holes Continuous backup to S3 (designed for 11 9s durability) Continuous scrubbing of data blocks Continuous monitoring of nodes and disks for repair 10 GB segments as unit of repair or hotspot rebalance to quickly rebalance load Quorum membership changes do not stall write operations AZ 1 AZ 2 AZ 3 Amazon S3
Traditional databases Have to replay logs since the last checkpoint Typically 5 minutes between checkpoints Single-threaded in MySQL; requires a large number of disk accesses Amazon Aurora Underlying storage replays redo records on demand as part of a disk read Parallel, distributed, asynchronous No replay for startup Checkpointed data Redo log Crash at T0 requires a reapplication of the SQL in the redo log since last checkpoint T0 T0 Crash at T0 will result in redo logs being applied to each segment on demand, in parallel, asynchronously Instant crash recovery
Survivable caches We moved the cache out of the database process Cache remains warm in the event of a database restart Lets you resume fully loaded operations much faster Instant crash recovery + survivable cache = quick and easy recovery from DB failures SQL Transactions Caching SQL Transactions Caching SQL Transactions Caching Caching process is outside the DB process and remains warm across a database restart
Faster, more predictable failover App RunningFailure Detection DNS Propagation Recovery Recovery DB Failure MYSQL App Running Failure Detection DNS Propagation Recovery DB Failure AURORA WITH MARIADB DRIVER 1 5 – 2 0 s e c . 3 – 2 0 s e c .
High availability with Aurora Replicas Amazon S3 AZ 1 AZ 2 AZ 3 Aurora primary instance Cluster volume spans 3 AZs Aurora Replica Aurora Replica db.r3.8xlarge db.r3.2xlarge Priority: tier-1 db.r3.8xlarge Priority: tier-0
High availability with Aurora Replicas Amazon S3 AZ 1 AZ 2 AZ 3 Aurora Primary instance Cluster volume spans 3 AZs Aurora Replica Aurora primary Instance db.r3.8xlarge db.r3.2xlarge Priority: tier-1 db.r3.8xlarge
ALTER SYSTEM CRASH [{INSTANCE | DISPATCHER | NODE}] ALTER SYSTEM SIMULATE percent_failure DISK failure_type IN [DISK index | NODE index] FOR INTERVAL interval ALTER SYSTEM SIMULATE percent_failure NETWORK failure_type [TO {ALL | read_replica | availability_zone}] FOR INTERVAL interval Simulate failures using SQL To cause the failure of a component at the database node: To simulate the failure of disks: To simulate the failure of networking:
Thank you! http://aws.amazon.com/rds/aurora

More Related Content

Deep Dive on Amazon Aurora

  • 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. July 13, 2016 Deep Dive on Amazon Aurora KD Singh, AWS Solution Architect Scott Ward, AWS Solution Architect
  • 2. MySQL-compatible relational database Performance and availability of commercial databases Simplicity and cost-effectiveness of open-source databases What is Amazon Aurora?
  • 3. Fastest growing service in AWS history Business applications Web and mobile Content management E-commerce, retail Internet of Things Search, advertising BI, analytics Games, media Aurora customer adoption
  • 4. Relational databases were not designed for cloud Multiple layers of functionality all in a monolithic stack SQL Transactions Caching Logging
  • 5. Not much has changed in the last 30 years Even when you scale it out, you’re still replicating the same stack SQL Transactions Caching Logging SQL Transactions Caching Logging Application SQL Transactions Caching Logging SQL Transactions Caching Logging Application SQL Transactions Caching Logging SQL Transactions Caching Logging Storage Application
  • 6. Reimagining the relational database What if you were inventing the database today? You wouldn’t design it the way we did in 1970 You’d build something  that can scale out …  that is self-healing …  that leverages existing AWS services …
  • 7. A service-oriented architecture applied to databases Moved the logging and storage layer into a multitenant, scale-out database-optimized storage service Integrated with other AWS services like Amazon EC2, Amazon VPC, Amazon DynamoDB, Amazon SWF, and Amazon Route 53 for control plane operations Integrated with Amazon S3 for continuous backup with 99.999999999% durability Control PlaneData Plane Amazon DynamoDB Amazon SWF Amazon Route 53 Logging + Storage SQL Transactions Caching Amazon S3 1 2 3
  • 8. SQL benchmark results 4 client machines with 1,000 connections each WRITE PERFORMANCE READ PERFORMANCE Single client machine with 1,600 connections Using MySQL SysBench with Amazon Aurora R3.8XL with 32 cores and 244 GB RAM
  • 9. Reproducing these results https ://d0.a wsstat ic . com /product -m ark eting/Aurora /R DS_ Auro ra_Perf orm ance_Assessm ent_Benchm ark ing_v 1-2 .pdf AMAZON AURORA R3.8XLARGE R3.8XLARGE R3.8XLARGE R3.8XLARGE R3.8XLARGE • Create an Amazon VPC (or use an existing one) • Create 4 EC2 R3.8XL client instances to run the SysBench client. All 4 should be in the same Availability Zone (AZ) • Enable enhanced networking on your clients • Tune Linux settings (see whitepaper referenced below) • Install SysBench version 0.5 • Launch a r3.8xlarge Amazon Aurora DB instance in the same VPC and AZ as your clients • Start your benchmark! 1 2 3 4 5 6 7
  • 10. Performance best practices MySQL/RDBMS practices still apply  Choose the right tool for the right job (OLAP vs. OLTP vs. NoSQL)  Create appropriate indexes  Tune your SQL code, use explain plans, performance schema  Many more… Leverage high concurrency  Aurora throughput increases with number of connections  Architect your applications to leverage high concurrency in Aurora Read scaling  Aurora offers read replicas with virtually no replication lag  Leverage multiple read replicas to distribute your reads
  • 11. Performance best practices Parameter tuning  No need to migrate your performance-related MySQL parameters to Aurora  Aurora parameter groups are pretuned and already optimal in most cases Performance comparison  Don’t obsess over individual metrics (CPU, IOPS, I/O throughput)  Focus on what matters, that is application performance Other best practices  Keep query cache on  Leverage Amazon CloudWatch metrics
  • 12. Advanced monitoring 50+ system/OS metrics | sorted process list view | 1–60 sec. granularity alarms on specific metrics | egress to CloudWatch Logs | integration with third-party tools ALARM
  • 13. Important systems and OS metrics User System Wait IRQ Idle Nice Steal CPU utilization Rx per declared ethn Tx per declared ethn Network Sleeping Running Total Stopped Blocked Zombie Processes Process ID Process name VSS Res Mem % consumed CPU % used CPU time Parent ID Process List Free Cached Buffered Total Writeback Inactive Dirty Mapped Slab Page tables Huge pages free Huge pages rsvd Huge pages surp Huge pages size Huge pages total Swap Swap free Swap committed Memory Read latency Write latency Read throughput Write throughput Read I/O/sec. Write I/O/sec. Queue depth Read queue depth Write queue depth Free local storage Device I/O Used Total Used Inodes/% Max Inodes/% File system 1 min 5 min 15 min Load Average
  • 14. Important database metrics  View database level metrics from Amazon Aurora and Amazon CloudWatch console  Perform retroactive workload analysis Select throughput Select latency DML throughput DML latency Commit throughput Commit latency DDL throughput DDL latency DB connections Active connections Login failures Buffer cache hit ratio Resultset cache hit ratio Deadlocks Blocked transactions Failed SQL statements Replica lag Replica lag maximum Replica lag minimum Free local storage
  • 15. Beyond benchmarks If only real-world applications saw benchmark performance POSSIBLE DISTORTIONS Real-world requests contend with each other Real-world metadata rarely fits in the data dictionary cache Real-world data rarely fits in the buffer cache Real-world production databases need to run at high availability
  • 16. Scaling user connections SysBench OLTP workload 250 tables Connections Amazon Aurora Amazon RDS MySQL 30 K IOPS (single AZ) 50 40,000 10,000 500 71,000 21,000 5,000 110,000 13,000 8x UP TO FA STER
  • 17. Scaling table count SysBench write-only workload 1,000 connections, default settings Tables Amazon Aurora MySQL I2.8XL local SSD MySQL I2.8XL RAM disk RDS MySQL 30 K IOPS (single AZ) 10 60,000 18,000 22,000 25,000 100 66,000 19,000 24,000 23,000 1,000 64,000 7,000 18,000 8,000 10,000 54,000 4,000 8,000 5,000 11x UP TO FA STER Number of write operations per second
  • 18. Scaling dataset size SYSBENCH WRITE-ONLY DB Size Amazon Aurora RDS MySQL 30 K IOPS (single AZ) 1 GB 107,000 8,400 10 GB 107,000 2,400 100 GB 101,000 1,500 1 TB 26,000 1,200 67x U P TO FA STER DB Size Amazon Aurora RDS MySQL 30K IOPS (single AZ) 80 GB 12,582 585 800 GB 9,406 69 CLOUDHARMONY TPC-C 136x U P TO FA STER
  • 19. Running with read replicas SysBench write-only workload 250 tables Updates per second Amazon Aurora RDS MySQL 30 K IOPS (single AZ) 1,000 2.62 ms 0 s 2,000 3.42 ms 1 s 5,000 3.94 ms 60 s 10,000 5.38 ms 300 s 500x UP TO LOW ER LA G
  • 20. Do fewer I/Os Minimize network packets Cache prior results Offload the database engine DO LESS WORK Process asynchronously Reduce latency path Use lock-free data structures Batch operations together BE MORE EFFICIENT How do we achieve these results? DATABASES ARE ALL ABOUT I/O NETWORK-ATTACHED STORAGE IS ALL ABOUT PACKETS/SECOND HIGH-THROUGHPUT PROCESSING DOES NOT ALLOW CONTEXT SWITCHES
  • 21. Aurora cluster Amazon S3 AZ 1 AZ 2 AZ 3 Aurora primary instance Cluster volume spans 3 AZs
  • 22. Aurora cluster with replicas Amazon S3 AZ 1 AZ 2 AZ 3 Aurora primary instance Cluster volume spans 3 AZs Aurora Replica Aurora Replica
  • 23. I/O traffic in RDS MySQL BINLOG DATA DOUBLE-WRITELOG FRM FILES T Y P E O F W R IT E MYSQL WITH STANDBY EBS mirrorEBS mirror AZ 1 AZ 2 Amazon S3 EBS Amazon Elastic Block Store (EBS) Primary Instance Standby Instance 1 2 3 4 5 Issue write to Amazon EBS—EBS issues to mirror, acknowledge when both done Stages write to standby instance using storage level replication Issues write to EBS on standby instance I/O FLOW Steps 1, 3, 5 are sequential and synchronous This amplifies both latency and jitter Many types of write operations for each user operation Have to write data blocks twice to avoid torn write operations OBSERVATIONS 780 K transactions 7,388 K I/Os per million transactions (excludes mirroring, standby) Average 7.4 I/Os per transaction PERFORMANCE 30 minute SysBench write-only workload, 100 GB dataset, RDS Single AZ, 30 K PIOPS
  • 24. I/O traffic in Aurora (database) AZ 1 AZ 3 Primary Instance Amazon S3 AZ 2 Replica Instance AMAZON AURORA ASYNC 4/6 QUORUM DISTRIBUTED WRITES BINLOG DATA DOUBLE-WRITELOG FRM FILES T Y P E O F W R IT E S 30 minute SysBench writeonly workload, 100GB dataset I/O FLOW Only write redo log records; all steps asynchronous No data block writes (checkpoint, cache replacement) 6x more log writes, but 9x less network traffic Tolerant of network and storage outlier latency OBSERVATIONS 27,378 K transactions 35x MORE 950K I/Os per 1M transactions (6x amplification) 7.7x LESS PERFORMANCE Boxcar redo log records—fully ordered by LSN Shuffle to appropriate segments—partially ordered Boxcar to storage nodes and issue write operations
  • 25. I/O traffic in Aurora (storage node) LOG RECORDS Primary Instance INCOMING QUEUE STORAGE NODE S3 BACKUP 1 2 3 4 5 6 7 8 UPDATE QUEUE ACK HOT LOG DATA BLOCKS POINT IN TIME SNAPSHOT GC SCRUB COALESCE SORT GROUP PEER TO PEER GOSSIPPeer Storage Nodes All steps are asynchronous Only steps 1 and 2 are in the foreground latency path Input queue is 46x less than MySQL (unamplified, per node) Favors latency-sensitive operations Use disk space to buffer against spikes in activity OBSERVATIONS I/O FLOW ① Receive record and add to in-memory queue ② Persist record and acknowledge ③ Organize records and identify gaps in log ④ Gossip with peers to fill in holes ⑤ Coalesce log records into new data block versions ⑥ Periodically stage log and new block versions to S3 ⑦ Periodically garbage-collect old versions ⑧ Periodically validate CRC codes on blocks
  • 26. Asynchronous group commits Read Write Commit Read Read T1 Commit (T1) Commit (T2) Commit (T3) LSN 10 LSN 12 LSN 22 LSN 50 LSN 30 LSN 34 LSN 41 LSN 47 LSN 20 LSN 49 Commit (T4) Commit (T5) Commit (T6) Commit (T7) Commit (T8) LSN GROWTH Durable LSN at head node COMMIT QUEUE Pending commits in LSN order TIME GROUP COMMIT TRANSACTIONS Read Write Commit Read Read T1 Read Write Commit Read Read Tn TRADITIONAL APPROACH AMAZON AURORA Maintain a buffer of log records to write out to disk Issue write operations when buffer is full, or time out waiting for write operations First writer has latency penalty when write rate is low Request I/O with first write, fill buffer till write picked up Individual write durable when 4 of 6 storage nodes acknowledge Advance DB durable point up to earliest pending acknowledgement
  • 27. Re-entrant connections multiplexed to active threads Kernel-space epoll() inserts into latch-free event queue Dynamically size threads pool Gracefully handles 5000+ concurrent client sessions on r3.8xl Standard MySQL—one thread per connection Doesn’t scale with connection count MySQL EE—connections assigned to thread group Requires careful stall threshold tuning CLIENTCONNECTION CLIENTCONNECTION LATCH FREE TASK QUEUE epoll() MYSQL THREAD MODEL AURORA THREAD MODEL Adaptive thread pool
  • 28. I/O traffic in Aurora (read replica) PAGE CACHE UPDATE Aurora Master 30% Read 70% Write Aurora Replica 100% New Reads Shared Multi-AZ Storage MySQL Master 30% Read 70% Write MySQL Replica 30% New Reads 70% Write SINGLE-THREADED BINLOG APPLY Data Volume Data Volume Logical: Ship SQL statements to replica Write workload similar on both instances Independent storage Can result in data drift between master and replica Physical: Ship redo from master to replica Replica shares storage; no writes performed Cached pages have redo applied Advance read view when all commits seen MYSQL READ SCALING AMAZON AURORA READ SCALING
  • 29. Availability “Performance only matters if your database is up”
  • 30. Storage node availability Quorum system for read/write; latency tolerant Peer-to-peer gossip replication to fill in holes Continuous backup to S3 (designed for 11 9s durability) Continuous scrubbing of data blocks Continuous monitoring of nodes and disks for repair 10 GB segments as unit of repair or hotspot rebalance to quickly rebalance load Quorum membership changes do not stall write operations AZ 1 AZ 2 AZ 3 Amazon S3
  • 31. Traditional databases Have to replay logs since the last checkpoint Typically 5 minutes between checkpoints Single-threaded in MySQL; requires a large number of disk accesses Amazon Aurora Underlying storage replays redo records on demand as part of a disk read Parallel, distributed, asynchronous No replay for startup Checkpointed data Redo log Crash at T0 requires a reapplication of the SQL in the redo log since last checkpoint T0 T0 Crash at T0 will result in redo logs being applied to each segment on demand, in parallel, asynchronously Instant crash recovery
  • 32. Survivable caches We moved the cache out of the database process Cache remains warm in the event of a database restart Lets you resume fully loaded operations much faster Instant crash recovery + survivable cache = quick and easy recovery from DB failures SQL Transactions Caching SQL Transactions Caching SQL Transactions Caching Caching process is outside the DB process and remains warm across a database restart
  • 33. Faster, more predictable failover App RunningFailure Detection DNS Propagation Recovery Recovery DB Failure MYSQL App Running Failure Detection DNS Propagation Recovery DB Failure AURORA WITH MARIADB DRIVER 1 5 – 2 0 s e c . 3 – 2 0 s e c .
  • 34. High availability with Aurora Replicas Amazon S3 AZ 1 AZ 2 AZ 3 Aurora primary instance Cluster volume spans 3 AZs Aurora Replica Aurora Replica db.r3.8xlarge db.r3.2xlarge Priority: tier-1 db.r3.8xlarge Priority: tier-0
  • 35. High availability with Aurora Replicas Amazon S3 AZ 1 AZ 2 AZ 3 Aurora Primary instance Cluster volume spans 3 AZs Aurora Replica Aurora primary Instance db.r3.8xlarge db.r3.2xlarge Priority: tier-1 db.r3.8xlarge
  • 36. ALTER SYSTEM CRASH [{INSTANCE | DISPATCHER | NODE}] ALTER SYSTEM SIMULATE percent_failure DISK failure_type IN [DISK index | NODE index] FOR INTERVAL interval ALTER SYSTEM SIMULATE percent_failure NETWORK failure_type [TO {ALL | read_replica | availability_zone}] FOR INTERVAL interval Simulate failures using SQL To cause the failure of a component at the database node: To simulate the failure of disks: To simulate the failure of networking: