Migrating Your Databases to AWS - Tools and Services.pdf

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Migrating Your Databases to AWS –
Tools and Services
Blair Layton
Business Development Manager
Database Services

What to Expect from the Session
• Migration Issues
• AWS Database Migration Service Overview
• AWS Schema Conversion Tool Overview
• When to use DMS and SCT
• Customer Success Stories
• Case Study – SQL Server on-prem to RDS PostgreSQL
• Migration Projects – Quick overview
• Q&A!

© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customers Want to Migrate to AWS, but…
They can’t afford long periods of application downtime
Tools that enable minimal downtime are expensive
It seems too complex and expensive to migrate
They still need a copy of the data on-premise
They want to migrate to an open source database
Sending large volumes of data to AWS requires an
expensive international network link
They don’t have the skills inside their organization

Traditional Approach to Migrate to AWS
1. Create your AWS account
2. Setup your Virtual Private Cloud (VPC) in AWS
3. Connect to AWS with a VPN or Direct Connect
4. Shutdown and backup your database
5. Transmit the backup to S3
6. Configure an EC2 instance with the DB software
7. Restore the backup
8. Configure EC2 instances for the application
9. Switch the users to use AWS

Traditional Approach to Migrate to AWS
1. Create your AWS account
2. Setup your Virtual Private Cloud (VPC) in AWS
3. Connect to AWS with a VPN or Direct Connect
4. Shutdown and backup your database
5. Transmit the backup to S3
6. Configure an EC2 instance with the DB software
7. Restore the backup
8. Configure EC2 instances for the application
9. Switch the users to use AWS
Steps 4-9 could take a week or more!

Now There is a Better Way!

AWS Database Migration Service (AWS DMS)
DMS migrates databases to AWS easily and
securely with minimal downtime. It can migrate
your data to and from most widely used
commercial and open-source databases.
Amazon Aurora

New NoSQL support
Migrate to AWS
• Move from MongoDB to Amazon DynamoDB
• Move from MongoDB to relational db’s
Move between NoSQL and SQL
• Change technologies
Amazon Aurora
DynamoDB
DynamoDB
RDS

New Support for S3 as a Source and Target
Extract Data from any supported DMS source to S3 and to any DMS target
S3 Bucket
Amazon Aurora
S3 Bucket Amazon Aurora
DynamoDB

S3 as a Source
Bulk Load
s3://mybucket/schemaName/tableName
s3://mybucket/hr/employee
/schemaName/tableName/LOAD001.csv
...
101,Smith,Bob,4-Jun-14,New York
102,Smith,Bob,8-Oct-15,Los Angeles
103,Smith,Bob,13-Mar-17,Dallas
CDC Load
s3://mybucket/changedata
CDC00001.csv
CDC00002.csv
CDC00003.csv
...
INSERT,employee,hr,101,Smith,Bob,4-Jun-14,New York
UPDATE,employee,hr,101,Smith,Bob,8-Oct-15,Los Angeles
UPDATE,employee,hr,101,Smith,Bob,13-Mar-17,Dallas
DELETE,employee,hr,101,Smith,Bob,13-Mar-17,Dallas

S3 as a Source – JSON External Table Definition
{
"TableCount": "1",
"Tables": [
{
"TableName": "employee",
"TablePath": "hr/employee/",
"TableOwner": "hr",
"TableColumns": [
{
"ColumnName": "Id",
"ColumnType": "INT8",
"ColumnNullable": "false",
"ColumnIsPk": "true"
},
{
"ColumnName": "LastName",
"ColumnType": "STRING",
"ColumnLength": "20"
},
…

S3 as a Target
Bulk Load
s3://mybucket/hr/employee
...
101,Smith,Bob,4-Jun-14,New York
102,Smith,Bob,8-Oct-15,Los Angeles
CDC Load
<time-stamp>.csv
<time-stamp>.csv
<time-stamp>.csv
...
I,101,Smith,Bob,4-Jun-14,New York
U,101,Smith,Bob,8-Oct-15,Los Angeles
U,101,Smith,Bob,13-Mar-17,Dallas
D,101,Smith,Bob,13-Mar-17,Dallas

Sources Databases for DMS
On-premises and Amazon EC2 instance databases:
• Oracle Database 10g–12c
• Microsoft SQL Server 2005–2016
• Azure SQL Database (CDC operations are not supported)
• MySQL 5.5–5.7
• MariaDB (MySQL-compatible data source)
• PostgreSQL 9.3+
• SAP ASE 15.7+
• MongoDB 2.6.x – 3.x
RDS instance databases:
• Microsoft SQL Server 2008R2–2016. CDC operations are not supported yet.
• MySQL versions 5.5–5.7
• PostgreSQL 9.4+. CDC only on 9.4.9+, 9.5.4+, 9.6+
• Amazon Aurora (MySQL-compatible data source)

Target Databases for DMS
On-premises and EC2 instance databases:
• Microsoft SQL Server 2005–2016
• MySQL 5.5–5.7
• PostgreSQL 9.3+
• SAP ASE 15.7+
RDS instance databases:
• Microsoft SQL Server 2008 R2 - 2016
• MySQL 5.5–5.7
• PostgreSQL 9.3+
• Amazon Aurora (MySQL-compatible data source)
• Amazon Redshift and Amazon DynamoDB

Customer
Premises
Application Users
AWS
Internet
VPN
Start a replication instance
Connect to source and target
databases
Select tables, schemas, or
databases
Let AWS DMS create tables,
load data, and keep them in
sync
Switch applications over to the
target at your convenience
Keep Your Apps Running During the Migration
AWS
Database Migration
Service

Multi-AZ Option for High Availability
Customer
Premises
or AWS
AWS
Internet
VPN
AWS
Database Migration
Service
AWS
Database Migration
Service

Change Data Capture (CDC) and Apply
Replication instance
Source Target
update
t1 t2
t1
t2
Transactions Change
apply
after bulk
load

Load is Table by Table
Source Target

Replication
instance
Source Target
Multiple Targets
Target
Target

Replication
instance
Source Target
Multiple Sources
Source
Source

Customers Don’t Have to Take Everything
Source Target
instance

Homogenous or Heterogeneous
Replication
instance
SQL Server MySQL
Replication
instance
Oracle Oracle
Replication
instance
Oracle Aurora

New Use Cases with DMS

New Use Cases with DMS
Migration of business critical applications
Migration from Classic to VPC
Cheap read replicas for Oracle
Read replicas on other engines
Cross region read replicas for Oracle and SQL Server
Analytics in the Cloud
Dev/Test and Production environment sync.

What Else Could You Do
With DMS?

AWS Database Migration Service Pricing
T2 for developing and periodic data migration tasks
C4 for large databases and minimizing time
T2 pricing starts at $0.028 per hour for T2.micro
C4 pricing starts at $0.213 per hour for C4.large
50 GB GP2 storage included with T2 instances
100 GB GP2 storage included with C4 instances
Data transfer inbound and within AZ is free
Data transfer across AZs starts at $0.01 per GB

Migrate 5 TB in 33 Hours!
In our testing, under mostly ideal conditions, we were able
to migrate 5 TB of relatively evenly distributed
data from a database on Amazon EC2 to a database on
Amazon RDS in about 33 hours. The data
included 4 large (250 GB) tables, a huge (1 TB) table, 1000
small to moderately sized tables, 3 tables
which contained LOBs varying between 25 GB and 75 GB,
and 10,000 very small tables.
- DMS Documentation

For less than $10 per TB!

My Database is Too Big!

Data Ingestion with AWS
AWS Import/Export Disk – Ship your hard disks
to AWS
AWS Import/Export Snowball – A secure
storage appliance with up to 80 TB that AWS
ships to you
Amazon S3 Transfer Acceleration – Use AWS
Edge Locations nearest to you to transfer data on
Amazon’s optimized network up to 300% faster.
You only need to pay for a local network
connection!

AWS Schema Conversion Tool (AWS SCT)
SCT helps automate many database schema and
code conversion tasks when migrating between
database engines or data warehouse engines
Amazon Aurora

AWS Schema Conversion Tool
Supports Schema Copy and RDS Recommendations

SCT Data Extractors
Extract Data from your data warehouse and migrate to Amazon Redshift
• Extracts through local migration agents
• Data is optimized for Redshift and Saved
in local files
• Files are loaded to an Amazon S3 bucket
(through network or Amazon Snowball)
and then to Amazon Redshift
Amazon
Redshift
AWS SCT S3 Bucket
• Greenplum Database (version 4.3 and later)
• Microsoft SQL Server (version 2008 and later)
• Netezza (version 7.0.3 and later)
• Oracle (version 10 and later)
• Teradata (version 13 and later)
• Vertica (version 7.2.2 and later)

SCT Helps with Converting Tables, Views & Code
Sequences
User-Defined Types
Synonyms
Packages
Stored Procedures
Functions
Triggers
Schemas
Tables
Indexes
Views
Sort and distribution keys

SCT can tell you how hard the migration will be
1. Connect SCT to
Source and Target
databases.
2. Run Assessment
Report.
3. Read Executive
Summary.
4. Follow detailed
instructions.

Pricing and Terms and Conditions
$0
for software license
Allowed Use
 Use SCT to migrate database schemas to
Amazon RDS, Amazon Redshift, or Amazon
EC2-based databases
 To use SCT to migrate schemas to other
destinations, contact for special pricing
Pricing
 Free software license
 For active AWS customers with
accounts in good standing

When to use DMS and SCT?

Strengths and Focus Areas
DMS can act as a replication/migration Swiss Army
knife, but is not a magic wand.
Use It
- Heterogeneous migrations
- Minimal downtime required
- No native solution
Caution
- Some tables with large LOBs
- Complex data types
- High load database
Don’t Use It
- Native no downtime solution exists
- Can take downtime + native
- > 5 TB + slow Internet****

Tools for Migration Project Phases
Phase Service/Tool Notes
Assessment AWS Schema Conversion Tool
Reports on the database objects, complexity and types of
migration issues
Schema Migration AWS Schema Conversion Tool
Copies a schema or migrates a schema depending on
whether it is a homogeneous or heterogeneous
migrations
Data Migration
AWS Database Migration Service
AWS Schema Conversion Tool
Bulk load and change data capture (CDC) options
Extraction and load for large data warehouses, including
AWS Snowball integration
Application Migration AWS Schema Conversion Tool SQL statement migration in application code
Data Validation Manual Count, Sum, or compare a hash of a query output
Functional Testing Various Tools on Marketplace Ensure the application runs as intended
Performance Testing Various Tools on Marketplace Ensure the application performance as intended

Tools for Migration Scenarios
Scenario Example Recommendation
Homogeneous
migration to the same
database version and
edition
Migration of Oracle Database
11gR2 Enterprise Edition from
on-premise to EC2
Use the native replication technology to
create a standby database and then failover
to the standby database
Homogeneous
migration to a different
version
Migration of MySQL 5.5 to
MySQL 5.7
AWS Schema Conversion Tool and AWS
Database Migration Service
Homogeneous
migration to a different
edition
Migration of SQL Server
Enterprise Edition to Standard
Edition
Heterogeneous
migration
Migration from Oracle
Database to PostgreSQL

Customer Successes

Heterogeneous migration
Oracle private DC to RDS PostgreSQL migration
Used the AWS Schema Conversion Tool to convert their
database schema
Used on-going replication (CDC) to keep databases in
sync until they reached the cutover window
Benefits:
• Improved reliability of the cloud environment
• Savings on Oracle licensing costs
• SCT Assessment Report let them understand the
scope of the migration

Scale-up migration
RDS MySQL to Amazon Aurora Migration
Used DMS with on-going replication (CDC) to
migrate the data
Benefits:
• Aurora handles their larger data storage
requirements. Per regulations they are storing
120 TB of data for 2 years
• Reduced cost and improved performance when
compared to large MySQL instances

Homogeneous migration
RDS MySQL on EC Classic to VPC
Database ran an end-user application so could
not take downtime
70 RDS instances have been migrated
Benefits:
• Leveraging CDC, they could decrease
outage per database to less than 5 mins
• 70 RDS instances have been seamlessly
migrated

Split migration
35 million members on it’s site
Many interdependent applications built over the
last 15 years with unique High IOPS
requirements
Migration from legacy cloud service to AWS
Combination of migration to MySQL on EC2 and
Amazon Aurora
Benefits:
• DMS reduced the time required to migrate our
databases by 40 percent
• Realized 55 percent cost savings by moving some
db’s to Amazon Aurora

Partner Case Study – Migration
from Microsoft SQL Server to
RDS PostgreSQL

On Premises Database
MSSQL Server 2008 R2 Enterprise Edition
Database Stats (April 2016):
• Number of Tables: 102
• Database Size: ~ 55GB
• Server Specs: 128GB RAM, Intel Xeon CPU
with 24 Cores
Company Profile
>2 millionApproximate Registered SSO Users
(June 2016)
~37Integrated Applications
(June 2016)
Ideal State
ASEAN Media Company
4.7 million residential customers
183 TV channels including 53 HD channels
Approx. 66% penetration of TV households
• database in AWS caters for SSO’s migration to the cloud
• able to manage current on-premises database load
• able to scale with the application
• must be highly available
• maintain cost efficiency while performing all of the above
Uberfusion Case Study – Data Migration for SSO Media

State of On-premise MSSQL Sever Database
Limited Scalability
As the on-premise MSSQL
Server Database is a
physical machine, scaling
requires purchasing and
installing additional
hardware.
Higher Costs
Hardware upgrades are
costly with a high hardware
delivery turn-around time.
Additionally, MSSQL
Server Enterprise Edition
licenses are high in costs.
Shared Infrastructure
The on-premises database
infrastructure is shared
across multiple
applications. A problem
caused by another
application could
potentially affect SSO.
New Mandate for SSO
To migrate the on-premise SSO system into Amazon Web
Services (AWS) cloud to reduce cost and improve
scalability.
Urgency
To be ready to handle
high load during Euro
and Olympics 2016
Migration into the Cloud
No data loss and completed in a short time frame at
minimal costs
The decision was made to go with RDS PostgreSQL, as:
- MSSQL on RDS does not support Multi Availability Zone
in AWS Singapore and hence cannot be highly available
- No need to maintain a VPN tunnel back to premise
- Lower OPEX costs compared to running MSSQL on
RDS or managing our own DB servers
Problem Statement, Mandate and Challenge
~11.5x
cheaper
Compared to RDS MSSQL
Server Enterprise

Uberfusion’s Delivery for Data Migration for SSO System
Challenge: Short Migration Window & Legacy Data
• Data cleansing was performed
• Instance created with access to both source and target database for
ease of migration
• Export scripts automated for speed and accuracy
• Multiple scripts to cater for special characters in data
• Logs offloaded from database to ElasticSearch and Kibana
Challenge: Leverage Managed Service
• Amazon’s RDS service for PostgreSQL was used
• System code base was updated to integrate with the new
database
Challenge: Testing
• Services were disabled pre-DNS migration for the SSO System
• Whitelist for specific users were created for testing and
verification of data
Challenge: Transparent to Users
• PostgreSQL Membership library was modified to mimic
ASP.NET Membership in regards to password encryption
• Migration was tested beforehand to ensure users would
still be able to login with their existing passwords
Amazon Elasticsearch
Service
Amazon Kinesis
Firehose
Challenge: Zero Data Loss
• All user-related data tables were identified
• System was put into read-only mode before starting migration
• Views were created in the schema of the new PostgreSQL database, and exported to flat
files
• Number of records were verified for pre- and post-migration
• Data was verified for accuracy from a sample of users post-migration
The Solution
Amazon RDS Postgre SQL instance
Postgre SQL instance
<1hr
Total Migration
Time
~1.5GB
Data size reduced
to
~1.8 millionSSO users migrated

Migration Steps
Disable
SSO
Service
Run script
to export
data to
flat files
according
to new
schema
Remove
headers
from
exported
files
Import
processed
files into
Amazon
PostgreSQ
L RDS
Test and
verify data
integrity
Create views to
match new schema
Re-enable SSO Service

Uberfusion’s Delivery for SSO Migration to AWS
Challenge: DB Migration from MSSQL to
Postgres
• Data exported into flat files with same
schema as new Postgres DB
• Membership libraries modified to
replicate same encryption in
PostgreSQL
> 500,000sessions in June 2016)
> 4x
TPS improvement
for login
Challenge: Scaling and Simplified Provisioning
• Separate applications on Amazon EC2
• Individual scaling groups according to load
Challenge: End-to-End Monitoring
• Silicone and service metrics monitored with Amazon
Cloudwatch
• Runscope used for API uptime and response time
monitoring
• Extensive logs input into Amazon Elastisearch via
Amazon Firehose, visualised via Kibana
50%
Page load
times
Challenge: Maintaining Connectivity
• Citrix Cloudbridge is leveraged to ensure
connectivity between AWS Cloud and on-
premise DC
Amazon EC2Auto Scaling Elastic Load Balancing AWS CodeDeploy CloudFormation
Amazon Elasticsearch
Service
Amazon Kinesis
FirehoseAmazon CloudWatch Amazon S3
Challenge: Improve System Performance
• Use of Redis cache instead of DB for sessions and time-based tickets
• Frequently accessed data is cached to reduce repeated DB reads
• Compression and Browser Caching enabled on Amazon CloudFront,
improving page load times by 50%
• TPS improved to 179 TPS for login on 4 auto-scaled servers vs 40 TPS
during on-premise
Amazon SES Amazon SQSAmazon ElastiCacheAmazon CloudFront Amazon S3
Fast Provisioning of Infrastructure
• Scales independently and only pay
for what you use.
• <1 day to duplicate environments vs
>5 days for on-premise.
The Benefits
Improved Development Efficiency
By leveraging AWS PaaS components,
man-days can be focused on other
parts of the system.
Improved Availability and System
Performance
• 50% improvement in page load
times.
Increased Visibility Allows for
Proactive Teams
• Detected McAfee FIM tool causing
CPU spikes.
• Detected iOS production application
hitting SSO Staging.
Sustain High Loads
• Anticipating Euro 2016 and
Olympics 2016 load.
• 179 TPS in AWS vs 40 TPS on-
premise.
The Solution and Benefits

Migration Projects

EXECUTE
AWS Migration Framework
• Project Control
⎼ Strategy (business driver)
⎼ Key Stakeholders and
Team
⎼ Plan (Scope, Schedule,
Resources)
⎼ Cost Estimation
• Portfolio discovery
• Migration plan
• Operations Integration
• Security
• Prioritized Backlog
⎼ Application groups
⎼ Migration strategy
⎼ Success criteria
• Ops Integration –
Foundation and Landing
Zone (target zone setup)
• Setup Factory (Tools,
Teams, Processes)
• Pilot Migration
Discover
Design
Build
Integrate
Validate
Cutover
Prioritized
Backlog
(PLAN)
• Application optimization
• Process optimization
• Operational optimization
• Cost optimization
READINESS AND
PLANNING
ACTIVATE
OPTIMIZE

AWS Migration Framework (Readiness &
Planning)
• Project Control
⎼ Strategy (business driver)
⎼ Key Stakeholders and
Team
⎼ Plan (Scope, Schedule,
Resources)
⎼ Cost Estimation
• Portfolio discovery
• Migration plan
• Operations Integration
• Security
READINESS AND
PLANNING
Discover, Assess &
PrioritizeApplications
Use Migration Tools
Transition Production
Redesign Application/
InfrastructureArchitecture
App Code
Development
PurchaseCOTS/
SaaS & licensing
Validation
Modify underlying
Infrastructure
Full ALM/ SDLC
Manual Config
Manual Deploy
Manual Install
Determine
Migration Path
Automate
Manual Install & Setup
Integration
Determine
new platform
The 6Rs of migration planning

For more information attend
the webinar:
Managing a Database Migration Project –
Best Practices and Customer Experiences

Remember to complete
your evaluations!

We hope you found it interesting! A kind reminder to
complete the survey and tell us your thoughts about today’s webinar
and how we can improve your webinar experience for you in the future
Thank You For Attending
AWS Cloud Migration Webinar Series

Migrating Your Databases to AWS - Tools and Services.pdf

More Related Content

Migrating Your Databases to AWS - Tools and Services.pdf