SlideShare a Scribd company logo

Deep Dive on Amazon RDS

•
•
Amazon Web Services
Amazon Web Services

RDS provides a fully managed relational database service. Key features include automated provisioning and scaling, high availability, data encryption at rest, backups and read replicas. RDS supports multiple database engines like MySQL, PostgreSQL, Oracle and SQL Server. Customers can migrate databases to RDS by backing up to S3 and restoring onto a new RDS instance.

1 of 53
Richard Ainley, Solutions Architect – AWS Deep Dive on Amazon Relational Database Service
Amazon Relational Database Service (RDS) No infrastructure management Scale up/down Cost-effective Instant provisioning Application compatibility
Trade-offs with a managed service Fully managed host and OS • No access to the database host operating system • Limited ability to modify configuration that is managed on the host operating system • No functions that rely on configuration from the host OS Fully managed storage • Max storage limits • Microsoft SQL Server—4 TB • MySQL, MariaDB, PostgreSQL, Oracle—6 TB • Aurora—64 TB • Growing your database is a process
Amazon RDS engines Commercial Open source Cloud native MySQL Compatible PostgreSQL Compatible Amazon Aurora
Security
Amazon Virtual Private Cloud (Amazon VPC) Securely control network configuration Availability Zone AWS Region 10.1.0.0/16 10.1.1.0/24 M
Amazon Virtual Private Cloud (Amazon VPC) Securely control network configuration Availability Zone AWS Region 10.1.0.0/16 10.1.1.0/24 Manage connectivity AWS Direct Connect VPN connection VPC peering Internet gateway Routing rules M
Security groups Database IP firewall protection Protocol Port Range Source TCP 3306 172.31.0.0/16 TCP 3306 “Application security group” Corporate address admins Application tier M
AWS IAM governed access You can use AWS Identity and Access Management (IAM) to: • Control who can perform actions on RDS • Authenticate to your RDS MySQL / Aurora DB • MySQL 5.6.34 / 5.7.16 or higher • Aurora 1.10 or higher • Not available for db.t1.micro / db.m1.small DBA and Ops RDS
Compliance Singapore MTCS 27001/9001 27017/27018
Compliance Aurora SOC 1, 2, 3 ISO 20001/9001 ISO 27107/27018 PCI HIPAA BAA MySQL SOC 1, 2, 3 ISO 20001/9001 ISO 27107/27018 PCI FedRamp HIPAA BAA UK Gov. Programs Singapore MTCS Oracle SOC 1, 2, 3 ISO 20001/9001 ISO 27107/27018 PCI FedRamp HIPAA BAA UK Gov. Programs Singapore MTCSMariaDB SOC 1, 2, 3 ISO 20001/9001 ISO 27107/27018 PCI PostgreSQL SOC 1, 2, 3 ISO 20001/9001 ISO 27107/27018 PCI UK Gov. Programs Singapore MTCS HIPAA BAA SQL Server SOC 1, 2, 3 ISO 20001/9001 ISO 27107/27018 PCI UK Gov. Programs Singapore MTCS
SSL Available for all six engines Database traffic encryption M
At Rest Encryption
Transparent Data Encryption AWS CloudHSM M Oracle Wallet M
At Rest Encryption for all RDS Engines AWS Key Management Service (KMS) Two-tiered key hierarchy using envelope encryption: • Unique data key encrypts customer data • AWS KMS master keys encrypt data keys • Available for ALL RDS engines Benefits: • Limits risk of compromised data key • Better performance for encrypting large data • Easier to manage small number of master keys than millions of data keys • Centralized access and audit of key activity Data key 1 Data key 2 Data key 3 Data key 4 Custom application Customer master key(s) Amazon RDS instance 3 Amazon RDS instance 2 Amazon RDS instance 1
Your RDS instance + Data key Encrypted data key Encrypted data Master key(s) in customer’s account AWS KMS 1. Launch your RDS instance 2. RDS instance requests encryption key to use to encrypt data, passes reference to master key in account 3. Client request authenticated based on permissions set on both the user and the key 4. A unique data encryption key is created and encrypted under the KMS master key 5. Plaintext and encrypted data key returned to RDS 6. Plaintext data key stored in memory and used to encrypt/decrypt RDS data How keys are used to protect your data M
Enabling encryption AWS Command Line Interface (AWS CLI) aws rds create-db-instance --region us-west-2 --db-instance-identifier sg-cli-test --allocated-storage 20 --storage-encrypted --db-instance-class db.m4.large --engine mysql --master-username myawsuser --master-user-password myawsuser aws rds create-db-instance --region us-west-2 --db-instance-identifier sg-cli-test1 -- allocated-storage 20 --storage-encrypted --kms-key-id xxxxxxxxxxxxxxxxxx --db- instance-class db.m4.large --engine mysql --master-username myawsuser --master-user- password myawsuser Console
Amazon RDS + AWS KMS useful tips • You can only encrypt on new database creation • Encryption cannot be removed • Master and read replica must be encrypted • Unencrypted snapshots can be turned into encrypted snapshots • Cannot restore MySQL to Aurora or Aurora to MySQL
Metrics and monitoring
Standard monitoring Amazon CloudWatch metrics for Amazon RDS  CPU utilization  Storage  Memory  Swap usage  DB connections  I/O (read and write)  Latency (read and write)  Throughput (read and write)  Replica lag  Many more Amazon CloudWatch Alarms  Similar to on-premises custom monitoring tools
Enhanced Monitoring Access to over 50 new CPU, memory, file system, and disk I/O metrics as low as 1 second intervals
Simplify monitoring with AWS Management Console Amazon Performance Insights for RDS  Database Load : Identifies database bottlenecks  Easy  Powerful  Identifies source of bottlenecks  Top SQL  Adjustable Time frame  Hour, day, week and longer Max CPU AWS re:Invent 2016 DAT206: https://youtu.be/ztmtJJTC8_Y?t=39m53s
High availability
Availability Zone A AWS Region 10.1.0.0/16 10.1.1.0/24 Availability Zone B 10.1.2.0/24 Synchronous Replication M S Single Availability Zone Configuration HA Multi Availability Zone Configuration
High availability—Multi-AZ to DNS Availability Zone A AWS Region Availability Zone B M S DNS mydatabase.us-west-2.rds.amazonaws.com:3306 M
Read Replicas Bring data close to your customer’s applications in different regions Relieve pressure on your master node for supporting reads and writes Promote a Read Replica to a master for faster recovery in the event of disaster Within or cross-region • MySQL • MariaDB • PostgreSQL • Aurora
High availability—Amazon Aurora storage • Storage volume automatically grows up to 64 TB • Quorum system for read/write; latency tolerant • Peer-to-peer gossip replication to fill in holes • Continuous backup to Amazon S3 (built for 11 9s durability) • Continuous monitoring of nodes and disks for repair • 10 GB segments as unit of repair or hotspot rebalance • Quorum membership changes do not stall writes AZ 1 AZ 2 AZ 3 Amazon S3
High availability—Amazon Aurora nodes • Aurora cluster contains primary node and up to 15 secondary nodes • Failing database nodes are automatically detected and replaced • Failing database processes are automatically detected and recycled • Secondary nodes automatically promoted on persistent outage, no single point of failure • Customer application can scale out read traffic across secondary nodes AZ 1 AZ 3AZ 2 Primary Node Primary Node Primary Node Primary Node Primary Node Secondary Node Primary Node Primary Node Secondary Node
Scaling on RDS
Why Scale? • Handle higher load or lower usage • Naturally grow over time • Control costs
What can I scale? Database Instance Read Replicas Storage
Amazon Aurora - Balanced Read Replica Access AZ 1 AZ 2 AZ 3 Aurora Primary instance Aurora Replica Aurora Replica Read Replica Endpoint
Scaling your instance up/down AWS Management Console
Scaling—single Availability Zone Availability Zone A AWS Region M S DNS mydatabase.us-west-2.rds.amazonaws.com:3306 MM
Scaling—Multi-AZ Availability Zone A AWS Region Availability Zone B M S DNS mydatabase.us-west-2.rds.amazonaws.com:3306 SS M
Scaling—automation AWS CLI Scheduled CLI—cron aws rds modify-db-instance --db-instance-identifier sg-cli-test --db-instance-class db.m4.large --apply-immediately #Scale down at 8:00 PM on Friday 0 20 * * 5 /home/ec2-user/scripts/scale_down_rds.sh #Scale up at 4:00 AM on Monday 0 4 * * 1 /home/ec2-user/scripts/scale_up_rds.sh
Scaling—automation Scheduled—AWS Lambda No server but still runs on a schedule! import boto3 client=boto3.client('rds') def lambda_handler(event, context): response=client.modify_db_instance(DBInstanceIdentifier='sg-cli-test', DBInstanceClass='db.m4.xlarge', ApplyImmediately=True) print response
Scaling—automation Metrics-based scaling • Amazon CloudWatch and AWS Lambda!
Scaling—automation import boto3 import json client=boto3.client('rds') def lambda_handler(event, context): message = event['Records'][0]['Sns']['Message'] parsed_message=json.loads(message) db_instance=parsed_message['Trigger']['Dimensions'][0]['value'] print 'DB Instance: ' + db_instance response=client.modify_db_instance(DBInstanceIdentifier=db_instance, DBInstanceClass='db.m4.large', ApplyImmediately=True) print response
Backups and snapshots
Automated Backups MySQL, PostgreSQL, MariaDB, Oracle, SQL Server • Scheduled daily volume backup of entire instance • Archive database change logs • 35-day retention • Taken from standby when running multi-AZ Aurora • Automatic, continuous, incremental backups • No impact on database performance • 35-day retention
How do automated RDS backups work? If database is Multi-AZ, the snapshot is taken from the standby =LatestRestorableTime Every day during your backup window, the RDS service creates a storage volume snapshot of your database Every five minutes, RDS backs up the transaction logs of your database
Restoring • Creates an entire new database instance • You define all the instance configuration, just like creating a new instance
Snapshots • Full copies of your RDS database • Independent of scheduled backups • Used to create a new RDS instance • Taken from the standby when running multi-AZ
Migrating onto RDS
MySQL Backup to Aurora via S3 Source MySQL Database Target Aurora Database Amazon S3 Use multi-part upload or S3 Transfer Acceleration for faster uploads AWS Import/Export Snowball
MySQL Backup to Aurora via S3
SQL Server Backup to RDS SQL Server via S3 Source SQL Server Database Target RDS Database Amazon S3 Use multi-part upload or S3 Transfer Acceleration for faster uploads AWS Import/Export Snowball
SQL Server Backup to RDS SQL Server via S3 Importing to RDS No Encryption Encryption exec msdb.dbo.rds_restore_database @restore_db_name=‘database_name’, @s3_arn_to_restore_from=‘arn:aws:s3:::bucket_name/file_name_and_extension’; exec msdb.dbo.rds_restore_database @restore_db_name=‘database_name’, @s3_arn_to_restore_from=‘arn:aws:s3:::bucket_name/file_name_and_extension’, @kms_master_key_arn=‘arn:aws:kms:region:account-id:key/key-id’;
 Move data to the same or different database engine  Keep your apps running during the migration  Start your first migration in 10 minutes or less  Replicate within, to, or from Amazon EC2 or RDS AWS Database Migration Service
 Migrate from Oracle and SQL Server  Move your tables, views, stored procedures, and data manipulation language (DML) to MySQL, MariaDB, and Aurora  Highlight where manual edits are needed AWS Schema Conversion Tool
Source Database Target Database on Amazon RDS Microsoft SQL Server (version 2008 and later) Amazon Aurora (MySQL or PostgreSQL), Microsoft SQL Server, MySQL, PostgreSQL MySQL (version 5.6 and later) Amazon Aurora (PostgreSQL), MySQL, PostgreSQL Oracle (version 10.2 and later) Amazon Aurora (MySQL or PostgreSQL), MySQL, Oracle, PostgreSQL PostgreSQL (version 9.1 and later) Amazon Aurora (MySQL), MySQL, PostgreSQL http://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/Welcome.html SCT supported OLTP conversions
Deep Dive on Amazon RDS

More Related Content

Deep Dive on Amazon RDS

  • 1. Richard Ainley, Solutions Architect – AWS Deep Dive on Amazon Relational Database Service
  • 2. Amazon Relational Database Service (RDS) No infrastructure management Scale up/down Cost-effective Instant provisioning Application compatibility
  • 3. Trade-offs with a managed service Fully managed host and OS • No access to the database host operating system • Limited ability to modify configuration that is managed on the host operating system • No functions that rely on configuration from the host OS Fully managed storage • Max storage limits • Microsoft SQL Server—4 TB • MySQL, MariaDB, PostgreSQL, Oracle—6 TB • Aurora—64 TB • Growing your database is a process
  • 4. Amazon RDS engines Commercial Open source Cloud native MySQL Compatible PostgreSQL Compatible Amazon Aurora
  • 6. Amazon Virtual Private Cloud (Amazon VPC) Securely control network configuration Availability Zone AWS Region 10.1.0.0/16 10.1.1.0/24 M
  • 7. Amazon Virtual Private Cloud (Amazon VPC) Securely control network configuration Availability Zone AWS Region 10.1.0.0/16 10.1.1.0/24 Manage connectivity AWS Direct Connect VPN connection VPC peering Internet gateway Routing rules M
  • 8. Security groups Database IP firewall protection Protocol Port Range Source TCP 3306 172.31.0.0/16 TCP 3306 “Application security group” Corporate address admins Application tier M
  • 9. AWS IAM governed access You can use AWS Identity and Access Management (IAM) to: • Control who can perform actions on RDS • Authenticate to your RDS MySQL / Aurora DB • MySQL 5.6.34 / 5.7.16 or higher • Aurora 1.10 or higher • Not available for db.t1.micro / db.m1.small DBA and Ops RDS
  • 11. Compliance Aurora SOC 1, 2, 3 ISO 20001/9001 ISO 27107/27018 PCI HIPAA BAA MySQL SOC 1, 2, 3 ISO 20001/9001 ISO 27107/27018 PCI FedRamp HIPAA BAA UK Gov. Programs Singapore MTCS Oracle SOC 1, 2, 3 ISO 20001/9001 ISO 27107/27018 PCI FedRamp HIPAA BAA UK Gov. Programs Singapore MTCSMariaDB SOC 1, 2, 3 ISO 20001/9001 ISO 27107/27018 PCI PostgreSQL SOC 1, 2, 3 ISO 20001/9001 ISO 27107/27018 PCI UK Gov. Programs Singapore MTCS HIPAA BAA SQL Server SOC 1, 2, 3 ISO 20001/9001 ISO 27107/27018 PCI UK Gov. Programs Singapore MTCS
  • 12. SSL Available for all six engines Database traffic encryption M
  • 15. At Rest Encryption for all RDS Engines AWS Key Management Service (KMS) Two-tiered key hierarchy using envelope encryption: • Unique data key encrypts customer data • AWS KMS master keys encrypt data keys • Available for ALL RDS engines Benefits: • Limits risk of compromised data key • Better performance for encrypting large data • Easier to manage small number of master keys than millions of data keys • Centralized access and audit of key activity Data key 1 Data key 2 Data key 3 Data key 4 Custom application Customer master key(s) Amazon RDS instance 3 Amazon RDS instance 2 Amazon RDS instance 1
  • 16. Your RDS instance + Data key Encrypted data key Encrypted data Master key(s) in customer’s account AWS KMS 1. Launch your RDS instance 2. RDS instance requests encryption key to use to encrypt data, passes reference to master key in account 3. Client request authenticated based on permissions set on both the user and the key 4. A unique data encryption key is created and encrypted under the KMS master key 5. Plaintext and encrypted data key returned to RDS 6. Plaintext data key stored in memory and used to encrypt/decrypt RDS data How keys are used to protect your data M
  • 17. Enabling encryption AWS Command Line Interface (AWS CLI) aws rds create-db-instance --region us-west-2 --db-instance-identifier sg-cli-test --allocated-storage 20 --storage-encrypted --db-instance-class db.m4.large --engine mysql --master-username myawsuser --master-user-password myawsuser aws rds create-db-instance --region us-west-2 --db-instance-identifier sg-cli-test1 -- allocated-storage 20 --storage-encrypted --kms-key-id xxxxxxxxxxxxxxxxxx --db- instance-class db.m4.large --engine mysql --master-username myawsuser --master-user- password myawsuser Console
  • 18. Amazon RDS + AWS KMS useful tips • You can only encrypt on new database creation • Encryption cannot be removed • Master and read replica must be encrypted • Unencrypted snapshots can be turned into encrypted snapshots • Cannot restore MySQL to Aurora or Aurora to MySQL
  • 20. Standard monitoring Amazon CloudWatch metrics for Amazon RDS  CPU utilization  Storage  Memory  Swap usage  DB connections  I/O (read and write)  Latency (read and write)  Throughput (read and write)  Replica lag  Many more Amazon CloudWatch Alarms  Similar to on-premises custom monitoring tools
  • 21. Enhanced Monitoring Access to over 50 new CPU, memory, file system, and disk I/O metrics as low as 1 second intervals
  • 22. Simplify monitoring with AWS Management Console Amazon Performance Insights for RDS  Database Load : Identifies database bottlenecks  Easy  Powerful  Identifies source of bottlenecks  Top SQL  Adjustable Time frame  Hour, day, week and longer Max CPU AWS re:Invent 2016 DAT206: https://youtu.be/ztmtJJTC8_Y?t=39m53s
  • 24. Availability Zone A AWS Region 10.1.0.0/16 10.1.1.0/24 Availability Zone B 10.1.2.0/24 Synchronous Replication M S Single Availability Zone Configuration HA Multi Availability Zone Configuration
  • 25. High availability—Multi-AZ to DNS Availability Zone A AWS Region Availability Zone B M S DNS mydatabase.us-west-2.rds.amazonaws.com:3306 M
  • 26. Read Replicas Bring data close to your customer’s applications in different regions Relieve pressure on your master node for supporting reads and writes Promote a Read Replica to a master for faster recovery in the event of disaster Within or cross-region • MySQL • MariaDB • PostgreSQL • Aurora
  • 27. High availability—Amazon Aurora storage • Storage volume automatically grows up to 64 TB • Quorum system for read/write; latency tolerant • Peer-to-peer gossip replication to fill in holes • Continuous backup to Amazon S3 (built for 11 9s durability) • Continuous monitoring of nodes and disks for repair • 10 GB segments as unit of repair or hotspot rebalance • Quorum membership changes do not stall writes AZ 1 AZ 2 AZ 3 Amazon S3
  • 28. High availability—Amazon Aurora nodes • Aurora cluster contains primary node and up to 15 secondary nodes • Failing database nodes are automatically detected and replaced • Failing database processes are automatically detected and recycled • Secondary nodes automatically promoted on persistent outage, no single point of failure • Customer application can scale out read traffic across secondary nodes AZ 1 AZ 3AZ 2 Primary Node Primary Node Primary Node Primary Node Primary Node Secondary Node Primary Node Primary Node Secondary Node
  • 30. Why Scale? • Handle higher load or lower usage • Naturally grow over time • Control costs
  • 31. What can I scale? Database Instance Read Replicas Storage
  • 32. Amazon Aurora - Balanced Read Replica Access AZ 1 AZ 2 AZ 3 Aurora Primary instance Aurora Replica Aurora Replica Read Replica Endpoint
  • 33. Scaling your instance up/down AWS Management Console
  • 34. Scaling—single Availability Zone Availability Zone A AWS Region M S DNS mydatabase.us-west-2.rds.amazonaws.com:3306 MM
  • 35. Scaling—Multi-AZ Availability Zone A AWS Region Availability Zone B M S DNS mydatabase.us-west-2.rds.amazonaws.com:3306 SS M
  • 36. Scaling—automation AWS CLI Scheduled CLI—cron aws rds modify-db-instance --db-instance-identifier sg-cli-test --db-instance-class db.m4.large --apply-immediately #Scale down at 8:00 PM on Friday 0 20 * * 5 /home/ec2-user/scripts/scale_down_rds.sh #Scale up at 4:00 AM on Monday 0 4 * * 1 /home/ec2-user/scripts/scale_up_rds.sh
  • 37. Scaling—automation Scheduled—AWS Lambda No server but still runs on a schedule! import boto3 client=boto3.client('rds') def lambda_handler(event, context): response=client.modify_db_instance(DBInstanceIdentifier='sg-cli-test', DBInstanceClass='db.m4.xlarge', ApplyImmediately=True) print response
  • 39. Scaling—automation import boto3 import json client=boto3.client('rds') def lambda_handler(event, context): message = event['Records'][0]['Sns']['Message'] parsed_message=json.loads(message) db_instance=parsed_message['Trigger']['Dimensions'][0]['value'] print 'DB Instance: ' + db_instance response=client.modify_db_instance(DBInstanceIdentifier=db_instance, DBInstanceClass='db.m4.large', ApplyImmediately=True) print response
  • 41. Automated Backups MySQL, PostgreSQL, MariaDB, Oracle, SQL Server • Scheduled daily volume backup of entire instance • Archive database change logs • 35-day retention • Taken from standby when running multi-AZ Aurora • Automatic, continuous, incremental backups • No impact on database performance • 35-day retention
  • 42. How do automated RDS backups work? If database is Multi-AZ, the snapshot is taken from the standby =LatestRestorableTime Every day during your backup window, the RDS service creates a storage volume snapshot of your database Every five minutes, RDS backs up the transaction logs of your database
  • 43. Restoring • Creates an entire new database instance • You define all the instance configuration, just like creating a new instance
  • 44. Snapshots • Full copies of your RDS database • Independent of scheduled backups • Used to create a new RDS instance • Taken from the standby when running multi-AZ
  • 46. MySQL Backup to Aurora via S3 Source MySQL Database Target Aurora Database Amazon S3 Use multi-part upload or S3 Transfer Acceleration for faster uploads AWS Import/Export Snowball
  • 47. MySQL Backup to Aurora via S3
  • 48. SQL Server Backup to RDS SQL Server via S3 Source SQL Server Database Target RDS Database Amazon S3 Use multi-part upload or S3 Transfer Acceleration for faster uploads AWS Import/Export Snowball
  • 49. SQL Server Backup to RDS SQL Server via S3 Importing to RDS No Encryption Encryption exec msdb.dbo.rds_restore_database @restore_db_name=‘database_name’, @s3_arn_to_restore_from=‘arn:aws:s3:::bucket_name/file_name_and_extension’; exec msdb.dbo.rds_restore_database @restore_db_name=‘database_name’, @s3_arn_to_restore_from=‘arn:aws:s3:::bucket_name/file_name_and_extension’, @kms_master_key_arn=‘arn:aws:kms:region:account-id:key/key-id’;
  • 50.  Move data to the same or different database engine  Keep your apps running during the migration  Start your first migration in 10 minutes or less  Replicate within, to, or from Amazon EC2 or RDS AWS Database Migration Service
  • 51.  Migrate from Oracle and SQL Server  Move your tables, views, stored procedures, and data manipulation language (DML) to MySQL, MariaDB, and Aurora  Highlight where manual edits are needed AWS Schema Conversion Tool
  • 52. Source Database Target Database on Amazon RDS Microsoft SQL Server (version 2008 and later) Amazon Aurora (MySQL or PostgreSQL), Microsoft SQL Server, MySQL, PostgreSQL MySQL (version 5.6 and later) Amazon Aurora (PostgreSQL), MySQL, PostgreSQL Oracle (version 10.2 and later) Amazon Aurora (MySQL or PostgreSQL), MySQL, Oracle, PostgreSQL PostgreSQL (version 9.1 and later) Amazon Aurora (MySQL), MySQL, PostgreSQL http://docs.aws.amazon.com/SchemaConversionTool/latest/userguide/Welcome.html SCT supported OLTP conversions