Database Migrations to the Cloud
- 1. Database Management and the Cloud
Knowledge is Power
Kellyn Pot’Vin-Gorman| Technical Intelligence Manager, Delphix
- 2. 2© 2017 Delphix Corporation
Kellyn Pot’Vin-Gorman
Technical Intelligence Manager for the Office
of CTO, Delphix
Multi-platform DBA, (Oracle, MSSQL, MySQL,
Sybase, Postgres…..)
Oracle ACE Director, (Alumni)
Oak Table Network
APEX Women in Technology Award, CTA 2014
STEM education with Raspberry Pi and Python
Liaison for Denver SQL Server User Group
Rocky Mountain Oracle Conference Director
and Board Director
Author, blogger, (http://dbakevlar.com)
- 3. 3© 2017 Delphix Corporation
Cloud Trends
85% of Enterprises have a multi-cloud strategy
77% are hybrid cloud, (different than the 2017 cloud survey)
Workloads are being run in the cloud-
41% in public clouds
38% in private clouds
Enterprise Companies are choosing cloud
65% want public cloud
63% want private cloud solutions
93% will be hybrid
State of the Cloud Survey, RightScale
- 4. 4© 2017 Delphix Corporation
This trend will only increase in the next five years as cloud
continues to overtake the industry
- 5. 5© 2017 Delphix Corporation
Source: Right Scale State of the Cloud 2016
0%
10%
20%
30%
40%
50%
60%
Physical Machine Virtual Machine Private Cloud Public Cloud
2017
2018
2019
Enterprise Company Cloud Migrations, 2017-2019
- 6. Cloud Adoption and How It’s Changing
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Public Cloud Private Cloud Hybrid Cloud Any Cloud
2016
2017
2018
http://www.rightscale.com/blog/cloud-industry-insights/cloud-computing-trends-2017-state-cloud-survey
- 7. Who’s Who
Azure’s market penetration
Up from 20% to 34%
AWS has remained flat, (data before S3 outage)
Google and Kubernetes is gaining interest in the market
Rackspace still owns some of the landscape
Oracle is making headway in the SaaS arena, (Software as a Service)
IBM? Archaic is the Future? MaaS, (Mainframe as a Service)
- 8. Cloud by Provider
Cloud Endure 2017
Trivia: Which Cloud
Vendor benefitted most
from the Amazon S3
outage?
Amazon
Microsoft
Google
IBM
Oracle
Other
- 9. Amazon Web Services
• Owning the space
• New innovations
• New Migrations tools
• New Performance tools
- 11. Google
CloudSQL = MySQL and PostgreSQL
Cloudspanner = RDBMS
Cloud Bigtable = NoSQL
Kubernetes is gold!
- 12. Oracle Cloud(s)
IaaS, PaaS and SaaS, oh my!
Bare Metal
Playing catchup
Trivia: What year
did Larry introduce
the Oracle Cloud?
- 16. 16© 2017 Delphix Corporation
Why Move to the Cloud?
Savings!!
Cost
Resource
Storage
Time
Well, that depends on….
Price per Gb
Complexity
SLAs
Minimizing Data
Footprint
Removing Data-in-
Flight
- 17. Sticker Shock
Estimates on Wasted Cloud Expenditures
Reports state that 30% of cloud is wasted
When measured, it was closer to 45%
Best Tool to Impact Waste?
Docker
Vmware Vsphere leads with 42% adoption
Azure Pack/Stack is private cloud that shows significant growth from 10-14%
Cloud vendors profit from vendor lock-in.
- 18. 18© 2017 Delphix Corporation
Just copy data and applications into the cloud…
Straightforward approach
Inefficient, non-incremental for large environments
Insecure, physical file transfers
A truck, (yes, I’m looking at you, “snowmachine”)
Archaic processes recommended by vendors
Start with backups to IaaS storage, then populate re-hosted applications
by restoring from those backups
Cloud backups are easy, known technology
How Are Companies Migrating to the Cloud?
- 19. 19© 2017 Delphix Corporation
Backups cloud service: from on-premises to the Cloud, then
restore- Assumption
Database server
Cloud Storage
Database server
- 20. 20© 2017 Delphix Corporation
Migration Complete?
Even if only development or test has been migrated to the
cloud…
Data is migrated, but this doesn’t count for ongoing data loads,
application connectivity across the network.
To refresh will take considerable time to perform.
Rarely a consideration for the difference in cost structure for
processing large amounts of data from on-premise to the cloud.
- 21. 21© 2017 Delphix Corporation
Network, the New Bottleneck
Few tools identify network latency in terms of percentage or
cause.
Explore potential latency between cloud services and data
transfer conditions on contracts.
Negotiate requirements before the agreement is signed and
data is handed over.
Many organization are investing in a dedicated WAN for cloud
environments and operations.
- 22. 22© 2017 Delphix Corporation
https://blogs.msdn.microsoft.com/brunoterkaly/2013/09/26/how-to-migrate-an-on-premises-sql-server-2012-database-to-windows-azure-sql-database/
How Microsoft Sees a Migration to Azure
What do I really
need?
- 23. 23© 2017 Delphix Corporation
Cost Estimates for Cloud Vendors - Azure
Storage Capacity LRS ZRS
First 1 TB / Month $0.024 per GB $0.03 per GB
Next 49 TB (1 to 50 TB) / Month $0.0236 per GB $0.0295 per GB
Next 450 TB (50 to 500 TB) / Month $0.0232 per GB $0.029 per GB
Next 500 TB (500 to 1,000 TB) / Month $0.0228 per GB $0.0285 per GB
https://azure.microsoft.com/en-us/pricing/details/storage/blobs/
- 24. 24© 2017 Delphix Corporation
Jeremiah Wilton, https://www.slideshare.net/AmazonWebServices/dat205-new-launch-introduction-to-aws-database-
migration-service
Amazon Web Services
- 25. 25© 2017 Delphix Corporation
https://www.slideshare.net/AmazonWebServices/dat205-new-launch-introduction-to-aws-database-migration-service
Cross Platform Migration from AWS
- 26. 26© 2017 Delphix Corporation
https://www.slideshare.net/AmazonWebServices/dat205-new-launch-introduction-to-aws-database-migration-service
Replicate data in on-premises databases to AWS
Replicate OLTP data to Amazon Redshift
Integrate tables from third-party software into your reporting or core
OLTP systems
Hybrid cloud is a stepping stone in migration to AWS
Amazon Cloud and AWS Choices
- 27. 27© 2017 Delphix Corporation
T2 pricing starts at $0.018 per Hour for T2.micro
C4 pricing starts at $0.154 per Hour for C4.large
50GB GP2 storage included with T2 instances
100GB GP2 storage included with C4 instances
Data transfer inbound and within AZ is free
Data transfer across AZs starts at $0.01 per GB
https://aws.amazon.com/ec2/pricing/
AWS Pricing
- 28. 28© 2017 Delphix Corporation
ProjectTimeline
Optimize
Migrate
Initial Testing
Requirements
Cloud Projects
- 29. 29© 2017 Delphix Corporation
What’s Missing?
Over 90% of cloud projects don’t
perform a preliminary identification of
latency or data issues that will impact
the project.
No strategy to identify or optimize BEFORE migrating to the cloud.
- 30. 30© 2017 Delphix Corporation
For Non-Production Systems…
Change the way you’ve always performed tasks.
Performing common tasks the same way as previously
might end up costing more.
Secure Data
All IaaS alternatives promote encryption for data in-flight
and for data at-rest, but encryption may not be the right
answer…
Secondary Considerations
- 31. 31© 2017 Delphix Corporation
If you don’t optimize for the cloud before you migrate..
Different cost structures will come back to bite you-
Much smaller storage footprint
Update data-transfers between systems/applications
Virtualize environments that have high percentage of
duplicate data
Why is there a Sticker Shock with the Cloud?
- 32. 32© 2017 Delphix Corporation
On Prem to the Cloud
Database server
Cloud Storage
Application
- 33. 33© 2017 Delphix Corporation
Old-Fashioned Migration to Cloud
Database server
Cloud Storage
Database server
Application
- 34. 34© 2017 Delphix Corporation
Traditional copy data management techniques
Developed without concerns about infrastructure chargeback
Nightly batchload ETLs
Traditional data pulls
Inefficient applications with thick database design
IaaS vendors monitor storage and data transfers
Help meet SLAs, garner profits
It’s not just the data that exists in the end, so transformations can equal
big money for cloud vendors.
Cost Structures?
- 35. 35© 2017 Delphix Corporation
Why Less Focus on Thin Provisioning
Storage costs
Thin-provisioning storage avoidance = $$$
Data transfer costs
Far less data transferred during provisioning/refresh operations
As opposed to the volume of data transferred using traditional
cloning techniques
Network-attached storage data transfers offset somewhat
Most non-production usage modifies or accesses only a tiny
proportion of the total database during lifetime
- 36. 36© 2017 Delphix Corporation
Optimized- on-premises into the Cloud
Source DB server
Oracle 9i, 10g, 11g, 12c
10TB storage
Delphix Virtualization Engine
8 TB storage
Target DB server
Oracle 10g, 11g, 12c
No database storage
Application
Optimized
- 37. 37© 2017 Delphix Corporation
Cloud and Performance
KNOW THY ENEMY…
Tune SQL and Apps to perform efficiently as possible- natural life of
database is growth, (in processes, resources, etc.) before migrating.
The less network latency, the better- network tracing to eliminate
database blame is important.
Many of the same tools and data provides value- The AWR,
(Automatic Workload Repository) is your friend!
Take advantage of cloud monitoring tools that can ease the learning
curve as you migrate to the cloud.
- 38. 38© 2017 Delphix Corporation
https://www.channele2e.com/2016/01/21/top-50-cloud-monitoring-and-management-tools-which-are-best/
1. Amazon CloudWatch
2. AppDynamics
3. Cloudmonix
4. Datadog
5. Dynatrace
6. Idera
7. Microsoft Cloud Monitoring
8. Nagios
9. New Relic
10.Solarwinds
The Best Cloud Monitoring and Management Tools,
(Alphabetically)
- 39. 39© 2017 Delphix Corporation
Amazon Cloudwatch
• This is a monitoring service for AWS
cloud Collect and track metrics
• Collect and monitor log files
• Set alarms, and automatically react
to changes
- 40. 40© 2017 Delphix Corporation
AppDynamics
https://docs.appdynamics.com/display/PRO14S/Architecture
- 41. 41© 2017 Delphix Corporation
Cloudmonix, (formerly AzureWatch)
• Supports Azure
• Supports both Windows
and secondary Linux with
Azure
• Agent is required on all
hosts.
- 42. 42© 2017 Delphix Corporation
Datadog
• Strong DevOps Deployment
• Docker, Jenkins compatible
• Support AWS, Google, IBM
• GUI can be overwhelming
- 43. 43© 2017 Delphix Corporation
Dynatrace
Manages both Linux and Windows hosts
AWS, Azure, Cloud Foundry, OpenStack, OpenShift and more.
Hybrid management console
- 44. 44© 2017 Delphix Corporation
Designed for service
providers, database
professionals and IT
professionals
Monitor and manage data
center environments.
Acquired CopperEgg platform
— now known as Uptime
Cloud Monitor.
Idera
- 45. 45© 2017 Delphix Corporation
Microsoft Cloud Monitoring • Microsoft Centric cloud
monitoring tools.
• Migration assistant and IIS
assistant
- 46. 46© 2017 Delphix Corporation
Nagios
• Has a network analyzer!
• Cloud support for Amazon, EC2/S3
• Offers log and application
performance monitoring
• Still has open source features
- 47. 47© 2017 Delphix Corporation
New Relic
• Multi-tenant sourced
• Monitoring
• AWS, Azure and Google
• Requires agent installation
- 49. 49© 2017 Delphix Corporation
All IaaS solutions provide encryption in-flight and encryption at-rest
- But encryption doesn’t protect data as much as it needs to be .
Europe already requires data masking, not just data encryption for any
confindential data:
- http://ec.europa.eu/justice/data-protection/article-
29/documentation/opinion-recommendation/files/2014/wp216_en.pdf
Confidential data
- 50. 50© 2017 Delphix Corporation
Confidential data
Exposure
Production
Non-production
- 51. 51© 2017 Delphix Corporation
Encryption is reversible data obfuscation, which is very different from
masking data.
Data masking is non-reversible.
It solves the issue at the data level.
Is authentication and authorization in non-production in compliance with
security goals?
All organizations will soon need to review if critical data in non-
production environments be accessible to developers, testers
and users.
Confidential data
- 52. 52© 2017 Delphix Corporation
Masking personally-identifiable, (PII, HIPPA, PCI, etc.) information
renders it useless from a security standpoint
Resolves both the technical and personal responsibility issue.
The data can be masked before it moves to non-production,
removing unnecessary risk.
Why Masking is Important-
- 53. 53© 2017 Delphix Corporation
Data virtualization to the Cloud
Delphix Virtualization Engine
8TB storage
Delphix Masking Engine
Delphix Virtualization Engine
8 TB storage
Target DB server
Oracle 10g, 11g, 12c
Linux Source
8 TB database
Linux Target
Oracle 10g, 11g, 12c
- 54. 54© 2017 Delphix Corporation
Confidential data
Exposure
Production
Non-production
Encryption
Mask
Solution
- 55. 55© 2017 Delphix Corporation
Virtualize and tune on-prem
Mask data before it’s in the cloud
Smaller footprint before its in flight.
It’s transformed by securely masking on-premises...
Then...
securely virtualized data can be safely replicated to IaaS
Target Complete Solutions to Ease Management and
Vulnerability
- 56. 56© 2017 Delphix Corporation
Watch for Chargebacks
Data in flight
Minimize data footprint
Use a cloud monitoring tool to simplify management.
Confidential Data
Data masking is often integrated with products and protects PCI,
PII, HIPPA and other confidential data.
Summary
Editor's Notes
- 1002 IT Professionals from Infrastructure and related technologies
Only 82% were multi-cloud the year before
Hybrid was only 71% the year before
- Notice the decline of on-prem physical and VMs
What are cloud machines, other than enhanced VMs??
- Only 1% of companies have no cloud strategies planned.
SaaS will be controlled, often by the original vendors that can lock customers into their cloud platform.
SAP = Hana. Azure = Microsoft, Linkedin, Slideshare…. Oracle=EBS, Peoplesoft, OBIEE. Amazon is taking large chunks out of each.
- Amazon and Azure own this race, but a price war could shake up the future of which cloud our data resides.
Almost 70% of the market is owned by Amazon and Microsoft- Azure doubles each year, up 93% in 2016, expected $20 billion for 2018
During Amazon S3 outage, it wasn’t Azure or Oracle that benefited, but Google, which increased its business over 12% in just two days.
- Taking over OOW and OTW…renting space and doing a lot of vendor sponsorships at events for Oracle, MSSQL and others!
- Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery.
Containers with GCP are a big deal and the cost effectiveness is impressive, along with Kubernetes
FaaS, Function as a Service or Serverless environments are a big deal for them.
Cloud SQL- MySQL and PostgreSQL investment
- Oct. 5th, 2011 was the year that Larry introduced the Oracle cloud.
Baremetal is all ODA’s
OPC is all Exalogics “knitted” together, private clouds are often exadatas and now the new ExaCM offering.
- They own it, they have customers invested and can more easily lock them in with upgrades and partial migrations to their cloud.
Private clouds are selling pretty high, but Exa is having difficulty proving the market and ODA has become the backbone for the Baremetal option to challenge Amazon.
- Still have some cloud hosts that they manage of their own.
More investing in technical experts to manage IaaS for companies.
- Amazon Pricing : https://aws.amazon.com/ec2/pricing/ Sep 2016
- This is a simple migration strategy.
- There are a ton more, but these are the ones I tested and found to be best of all paid and open source for monitoring.
There are licenses and fees with many of these.
- Not able to execute powershell commands without agent installation on Azure, but agent is not required for standard monitoring.
- Well-known for a range of monitoring and management tools.
- Or does it shift the problem toward authentication and authorization?