Getting Started with Amazon Aurora

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Puneet Agarwal, Solution Architect, AWS
Steve Loyd, VP, Engineering Operations, Zendesk
Kalyan Wunnava, Manager, Database Engineering, Zendesk
August 11th, 2016
Getting Started with Amazon Aurora

What to expect from the session
• Introduction to Amazon RDS
• Why AWS built Aurora
• Customer adoption
• Aurora features
• Migrations and pricing
• Welcome Zendesk

Options for hosting databases
Self-managed Amazon EC2 instances Fully managed
Corporate data
center
Database DB on EC2
instance RDS

If you host your databases on premises
Power, HVAC, net
Rack and stack
Server maintenance
OS patches
DB s/w patches
Database backups
High availability
DB s/w installs
OS installation
you
Scaling
App optimization

Power, HVAC, net
Rack and stack
Server maintenance
OS patches
DB s/w patches
Database backups
Scaling
High availability
DB s/w installs
OS installation
you
App optimization
If you host your databases on premises
?

If you host your databases in EC2
Power, HVAC, net
Rack and stack
Server maintenance
OS patches
DB s/w patches
Database backups
Scaling
High availability
DB s/w installs
OS installation
you
App optimization

OS patches
DB s/w patches
Database backups
Scaling
High availability
DB s/w installs
you
App optimization
Power, HVAC, net
Rack and stack
Server maintenance
OS installation
If you host your databases in EC2
?

If you choose a managed database service
Power, HVAC, net
Rack and stack
Server maintenance
OS patches
DB s/w patches
Database backups
App optimization
High availability
DB s/w installs
OS installation
you
Scaling
Database tuning
Design consultation
App optimization
Best practices

Relational databases
Fully managed and secure
Fast, predictable performance
Simple and fast to scale
Low cost, pay for what you use
Amazon
RDS
Amazon Aurora

Why AWS built Amazon Aurora
 Speed and availability of high-end commercial databases
 Simplicity and cost-effectiveness of open source databases
 Drop-in compatibility with MySQL
 Simple pay as you go pricing
Delivered as a managed service

Database architectures in last 30 years
Even when you scale it out, you’re still replicating the same stack
SQL
Transactions
Caching
Logging
SQL
Transactions
Caching
Logging
Application
SQL
Transactions
Caching
Logging
SQL
Transactions
Caching
Logging
Application
SQL
Transactions
Caching
Logging
SQL
Transactions
Caching
Logging
Storage
Application

A service-oriented architecture applied to the database
Moved the logging and storage layer into a
multitenant, scaled-out database-optimized
storage service
Integrated with other AWS services like
Amazon EC2, Amazon VPC, Amazon
DynamoDB, Amazon SWF, and Amazon
Route 53 for control plane operations
Integrated with Amazon S3 for continuous
backup with 99.999999999% durability
Control planeData plane
Amazon
DynamoDB
Amazon SWF
Amazon Route 53
Logging + Storage
SQL
Transactions
Caching
Amazon S3
1
2
3

Rapid adoption of Amazon Aurora

Fastest growing service
in AWS history
Aurora customer adoption

Expedia: On-line travel marketplace
 Real-time business intelligence and analytics on a
growing corpus of online travel marketplace data.
 Current Microsoft SQL Server–based architecture
is too expensive. Performance degrades as data
volume grows.
 Cassandra with Solr index requires large memory
footprint and hundreds of nodes, adding cost.
Aurora benefits:
 Aurora meets scale and performance
requirements with much lower cost.
 25,000 inserts/sec with peak up to 70,000. 30 ms
average response time for write and 17 ms for
read, with 1 month of data.
World’s leading online travel
company with a portfolio that includes
150+ travel sites in 70 countries.

ISCS: Insurance claims processing
 Have been using Oracle and SQL Server for
operational and warehouse data.
 Cost and maintenance of traditional commercial
database has become the biggest expenditure and
maintenance headache.
Aurora benefits:
 The cost of a “more capable” deployment on Aurora
has proven to be about 70% less than ISCS’s SQL
Server deployments.
 Eliminated backup window with Aurora’s continuous
backup; exploiting linear scaling with number of
connections; continuous upload to Amazon Redshift
using Aurora Replicas.
Provides policy management,
claim, and billing solutions for
casualty and property insurance
organizations.

“When we ran Alfresco’s workload on Aurora, we were blown away to find that
Aurora was 10 times faster than our MySQL environment,” said John Newton,
founder and CTO of Alfresco. “Speed matters in our business, and Aurora has
been faster, cheaper, and considerably easier to use than MySQL.”
Amazon Aurora is fast

• Four client machines with 1,000 threads each
WRITE PERFORMANCE READ PERFORMANCE
• Single client with 1,600 threads
• MySQL SysBench
• R3.8XL with 32 cores and 244 GB RAM
SQL benchmark results

Aurora performance
• Writes scale with connection count
• Consistent performance as table count increases
• Consistent performance with growing datasets
• Minimal replica lag with high update frequency on
master (~5.38 ms with 10k updates per second)

Amazon Aurora is highly available

Amazon Aurora is highly available
Highly available storage
• Six copies of data
across three AZs
• Latency tolerant
quorum system for
read/write
• Up to 15 replicas with
low replication lag
Survivable caches
• Cache remains warm
in the event of a
database restart
• Lets you resume fully
loaded operations
much faster
Instant crash recovery
• Underlying storage
replays redo records
on demand as part of a
disk read
• Parallel, distributed,
asynchronous
AZ 1 AZ 2 AZ 3
Amazon
S3
SQL
Transactions
Caching
T0

Choose cross-region read replicas for faster disaster
recovery and enhanced data locality
Promote a read replica to a
master for faster recovery in the
event of disaster
Bring data close to your
customer’s applications in
different regions
Promote to a master for easy
migration

Amazon Aurora is easy to use
“Amazon Aurora’s new user-friendly monitoring interface made it
easy to diagnose and address issues. Its performance, reliability, and
monitoring really shows Amazon Aurora is an enterprise-grade AWS
database.” —Mohamad Reza, information systems officer at United
Nations

Simplify storage management
• Automatic storage scaling up to 64 TB—no performance impact
• Continuous, incremental backups to Amazon S3
• Instantly create user snapshots—no performance impact
• Automatic restriping, mirror repair, hot spot management, encryption
Up to 64 TB of storage—autoincremented in 10 GB units
up to 64 TB

Simplify monitoring with AWS Management Console
Amazon CloudWatch
metrics for RDS
 CPU utilization
 Storage
 Memory
 50+ system/OS metrics
 1–60 second granularity
 DB connections
 Selects per second
 Latency (read and write)
 Cache hit ratio
 Replica lag
CloudWatch alarms
 Similar to on-premises custom
monitoring tools

Simplify data security
 Encryption to secure data at rest
• AES-256; hardware accelerated
• All blocks on disk and in Amazon S3 are encrypted
• Key management by using AWS KMS
 SSL to secure data in transit
 Network isolation by using Amazon VPC by
default
 No direct access to nodes
 Supports industry standard security and data
protection certifications
Storage
SQL
Transactions
Caching
Amazon S3
Application

Migration from MySQL
Source database on RDS
 Snapshot migration: One-click
migration from RDS MySQL 5.6 to
Aurora
Source database external or on
EC2
 Use native MySQL migration tools
 Back up to S3 using Percona
XtraBackup, restore from S3
DB
snapshot
One-click
migrate
RDS MySQL
master/slave New Aurora
cluster
RDS snapshot migration

Start your first migration in 10 minutes or less
Keep your apps running during the migration
Replicate within, to, or from Amazon EC2 or RDS
Move data to the same or different database
engine
AWS
Database Migration
Service

Customer
premises
Application users
AWS
Internet
VPN
Start a replication instance
Connect to source and target databases
Select tables, schemas, or databases
Let AWS Database Migration Service
create tables, load data, and keep
them in sync
Switch applications over to the target
at your convenience
Keep your apps running during the migration
AWS
Database Migration
Service

Migrate from Oracle and SQL Server
Move your tables, views, stored procedures, and
data manipulation language (DML) to MySQL,
MariaDB, and Amazon Aurora
Know exactly where manual edits are needed
Download at aws.amazon.com/dms
AWS
Schema Conversion
Tool

Know exactly where manual edits are needed

Simple pricing
No licenses
No lock-in
Pay only for what you use
Discounts
44% with a 1-year Reserved Instance
63% with a 3-year Reserved Instance
vCPU Mem Hourly Price
db.r3.large 2 15.25 $0.29
db.r3.xlarge 4 30.5 $0.58
db.r3.2xlarge 8 61 $1.16
db.r3.4xlarge 16 122 $2.32
db.r3.8xlarge 32 244 $4.64
• Storage consumed, up to 64 TB, is $0.10/GB-month
• IOs consumed are billed at $0.20 per million I/O
• Prices are for US East (N. Virginia) region
Enterprise grade, open source pricing

Cost of ownership: Aurora vs. MySQL
MySQL configuration hourly cost
Primary
r3.8XL
Standby
r3.8XL
Replica
r3.8XL
Replica
R3.8XL
Storage
6TB/10K PIOP
Storage
6TB/10K PIOP
Storage
6TB/5K PIOP
Storage
6TB/5K PIOP
$3.78/hr
$3.78/hr
$3.78/hr $3.78/hr
$2.42/hr
$2.42/hr $2.42/hr
Instance cost: $15.12/hr
Storage cost: $8.30/hr
Total cost: $23.42/hr
$2.42/hr

Aurora configuration hourly cost
Total cost: $18.35/hr
Primary
r3.8XL
Replica
r3.8XL
Replica
R3.8XL
Storage/6 TB
$4.64/hr $4.64/hr $4.64/hr
$4.43/hr
*At a macro level, Aurora saves over 50% in
storage cost compared to RDS MySQL.
21.6%
Savings
 No idle standby instance
 Single shared storage volume
 No PIOPS—pay for use IO
 Reduction in overall IOP

Further opportunity for saving
Total cost: $11.39/hrStorage IOPS assumptions:
1. Average IOPS is 50% of maximum IOPS
2. 50% savings from shipping logs vs. full pages
51.3%
Savings
Primary
r3.8XL
Replica
r3.8XL
Replica
r3.8XL
Storage/6 TB
$2.32/hr $2.32/hr $2.32 hr
$4.43/hr
r3.4XL r3.4XL r3.4XL
 Use smaller instance size
 Pay-as-you-go storage

Speaker Name
Steve Loyd
Vice President, Engineering Operations
Speaker Name
Kalyan Wunnava
Database Engineering Manager

Zendesk is software for better customer service.

Growing global business
Sales, marketing, and customer success
teams
81,000
Global customers
SAN FRANCISCO
LONDON
COPENHAGEN
MELBOURNE
DUBLIN
TOKYO
MADISON
MANILA
SÅO PAULO
BERLIN
SINGAPORE
NEW YORK
PARIS
54%
Year over year revenue growth

Multiproduct platform
SUPPORT VOICE CHAT
ANALYTICSMESSAGE

Security, compliance, and customer success for today’s enterprise needs*
Compliance
SOC II Type 2
ISO 27001
ISO 27018
HIPAA
PCI (Q3 2016)
Security
Data encryption in transit/rest
Two-factor authentication
Automatic credit card redaction
SSO SAML support
Redundancy and disaster reco
Global data centers (colocation and AWS)
Success and support
Professional services
Priority support
Service-level agreements
Training and certification
Customer success strategists
*Not all features listed are currently available on all products

• Primary data store for key information since the beginning
• Three core uses: Account, alloc, sharded (customer)
• Sharded data store has seen the most growth year over year
• Over time we’ve added many other primary and secondary stores, but this
relational data store is still key for much of our data
MySQL data store importance

• Since 2006 we’ve faced a number of challenges in scaling our MySQL datastore
Growth challenges
2006
2008
2010
2012
2014
2016
Huge customer
add pushes us to
NvRAM drives
Growth of
customers leads
to sharding for all
customer data
Implementation
of zero downtime
tool for master
failover
Account move effort
leads to
development of zero
downtime tool for
indiv account moves
Flexible features = query perf challenges
Increasing maintenance cost of a growing data store platform
Interest in Aurora
develops

• Zendesk has been a user of AWS for past five
years, but in very specific uses.
• 2015
• Made our way to Vegas and learned all we
could
• Kicked off 120-day project to build full
Zendesk stack in AWS
• Succeeded and learned along the way
• While our stack now includes more than 40
host groups, Amazon Aurora was a big part
of our success
2015 – Zendesk’s first full AWS deployment
Image: https://jumpcloud.com/blog/wp-content/uploads/2014/11/AWS-in-vegas.jpg

Zendesk database environment – An overview
• 7 regions: 4 on premises and 3 on AWS
• Percona 5.6 and Amazon Aurora
• MySQL DBs: Shards (for customer data), account, and ID generation for shards
• MySQL storage on Fusion-io and NVMe for on-premises
Shards (DB cluster)
• Master and 2 slaves
• Single MySQL instance on a server
• Multiple shards on each server with identical schema
• Multiple customers on each shard
• Customer data on one shard only

Why Zendesk chose AWS Aurora
• Rapid provisioning compared to on-premises
• MySQL compatibility
• Managed environment with solid backups
• Elasticity – scale up/down with a button push
• Built-in support for encryption at rest
Aurora over RDS MySQL
• Cost effectiveness
• Database growth up to 64 TB
• Scalability to 15 replicas
• Better performance tests results

Limitation (RDS) Workaround
MyISAM engine not supported (Aurora) Use RDS MySQL which supports MyISAM
No DB server access (RDS) Rewrite scripts to run from client side
No physical access to binary logs (RDS)
Use MySQL commands to read binlog events
(e.g. Zendesk’s accounts migration tool)
SUPER MySQL privilege not available (RDS) Redefine grants (skip SUPER)
Limitations (in RDS) / workarounds

Limitation (RDS) Workaround
Can’t run SET GLOBAL commands for dynamic
config changes
Adjust config params in cluster/instance parameter
group
Can’t run STOP/START SLAVE or CHANGE
MASTER
Use RDS supplied system stored procs
Can’t turn OFF binary logging (global/session) Not available!
COMPRESSED row format not supported
Aurora assumes COMPACT in the back end
(Schema definition still conforms to original; no
schema drift)
Limitations (in RDS) / workarounds

Example of COMPRESSED row format table
mysql> create table test1
-> (
-> id INT NOT NULL PRIMARY KEY AUTO_INCREMENT
-> ,column1 TEXT NOT NULL
-> )
-> ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPRESSED
-> ;
Query OK, 0 rows affected, 2 warnings (0.05 sec)
mysql> show warnings;
+---------+------+--------------------------------------------------+
| Level | Code | Message |
+---------+------+--------------------------------------------------+
| Warning | 1478 | ROW_FORMAT=COMPRESSED is not currently supported |
| Warning | 1478 | InnoDB: assuming ROW_FORMAT=COMPACT. |
+---------+------+--------------------------------------------------+
mysql> show create table test1G
Create Table: CREATE TABLE `test1` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`column1` text NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 ROW_FORMAT=COMPRESSED

Summary and Zendesk next steps...
• Aurora has been a key part of our AWS expansion, allowing for fast
implementation and an easy-to-scale infrastructure
• We want to complete changes to our app to make use of replicas
• Already working to build out two additional “pods” of our services in new
regions!

Remember to complete
your evaluations!

Getting Started with Amazon Aurora

Related slideshows

More Related Content

Getting Started with Amazon Aurora