데이터 마이그레이션 AWS와 같이하기 - 김일호 솔루션즈 아키텍트:: AWS Cloud Track 3 Gaming
•
3 likes•1,331 views
This document discusses various options for migrating data and workloads between on-premises environments and AWS. It covers tools like AWS Database Migration Service for database migration, VM Import/Export for virtual machine migration, copying files between S3 buckets, and using services like Route53 for transitioning traffic during a migration. Specific techniques discussed include copying AMIs, EBS snapshots, security groups, and database parameters between regions; using the AWS Schema Conversion Tool; and DynamoDB cross-region replication.
Report
Share
데이터 마이그레이션 AWS와 같이하기 - 김일호 솔루션즈 아키텍트:: AWS Cloud Track 3 Gaming
- 1. 데이터 마이그레이션 AWS와 같이 하기 김일호 | Solutions Architect
- 2. AWS Global Infrastructure yesterday Region Edge Location 11 Regions 30 Availability Zones 54 Edge Locations Need to update
- 3. AWS Global Infrastructure TODAY Region Edge Location 12 Regions 32 Availability Zones 54 Edge Locations Need to update Seoul Region launched
- 4. Design Options on AWS Multi-AZ Cross Region Hybrid IT Single AZ AZ AZ AZ Region Region
- 5. Design Options on AWS Multi-AZ Cross Region Hybrid IT Single AZ AZ AZ AZ Region Region Compute
- 6. A example architecture on AWS RDS DB Instance Active (Multi-AZ) Availability Zone ELB Balancer Amazon S3 Amazon CloudFront Amazon Route 53 User Web Instances
- 7. Amazon DynamoDB Amazon DynamoDB Ap-southeast-1 Ap-northeast-1 AWS Data Pipeline AMI Amazon EBS AMI Copy Copy Copy RDS SnapshotAmazon RDS Snapshot Amazon EBSAmazon EBS SnapshotAmazon EBS Snapshot Snapshot Restore EC2 EC2 RestoreCreate Snapshot Restore Copy AWS provided cross-region copies
- 8. Migrating compute and network – EC2 instance § Resources § SSH Keys § Security Groups § AMI § EBS Volumes § Elastic IP Addresses § ELB § Launch configurations and AutoScaling group § VPC § Considerations § Reserved Instances § Services Limits
- 9. AMI Copy
- 10. Migrating compute and network – EC2 instance 1. Run Copy AMI at console 2. Check the status of copy AMI across region 3. Launch a new instance in target region
- 11. Migrating compute and network – EC2 instance 1. Create EBS volume snapshot à Copy Snapshot 2. Check the status of Snapshot copy cross region 3. Create a new volume in target region
- 12. Migrating compute and network – EC2 instance § Collect security group configuration with AWS CLI § $ ec2-describe-group -H --region ap-northeast-1 >> sg.txt
- 13. VM Import/Export
- 14. What is VM Import/Export http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/UsingVirtualMachinesinAmazonEC2.html
- 15. VM Import/Export will help §Migrate existing applications and workloads to Amazon EC2 §Copy your VM image catalog to Amazon EC2 §Create a disaster recovery repository for VM images
- 16. VM Import/Export steps 1. Install the AWS CLI. For more information. 2. Prepare the VM for import to Amazon EC2. 3. Export the VM from the virtualization environment. 4. Import the VM into Amazon EC2. For information. Launch the instance in Amazon EC2.
- 17. § Microsoft Windows Server 2003 (Standard, Datacenter, Enterprise) with Service Pack 1 (SP1) or later § Microsoft Windows Server 2003 R2 (Standard, Datacenter, Enterprise) § Microsoft Windows Server 2008 (Standard, Datacenter, Enterprise) § Microsoft Windows Server 2008 R2 (Standard, Datacenter, Enterprise) § Microsoft Windows Server 2012 (Standard, Datacenter) § Microsoft Windows Server 2012 R2 (Standard, Datacenter) § Red Hat Enterprise Linux (RHEL) 5.1- 5.11, 6.1-6.6, 7.0-7.1 § SUSE Linux Enterprise Server 11-12 § Ubuntu 12.04, 12.10, 13.04, 13.10, 14.04, 14.10, 15.04 § CentOS 5.1-5.11, 6.1-6.6, 7.0-7.1 § Debian 6.0.0-6.0.8, 7.0.0-7.8.0, 8.0.0 § Oracle Enterprise Linux 6.1-6.6, 7.0- 7.1 § Fedora Server 19-21 OSs supported by VM Import/Export Note: From 21-Dec-2015 Document You can import Windows and Linux VMs that use VMware ESX or Workstation, Microsoft Hyper-V, and Citrix Xen virtualization formats
- 18. Design Options on AWS Multi-AZ Cross Region Hybrid IT Single AZ Files and Objects AZ AZ AZ Region Region Compute
- 19. COPY Files from IDC to S3
- 20. Copy files from IDC to S3 Corporate Data Center Amazon S3 Bucket Storage Servers Files File systems
- 22. Think Parallel 1 File 2 Seconds 1 File 2 Seconds 1 File 2 Seconds 1 File 2 Seconds 8 Seconds Foreach($file in $files) {Write-S3Object -BucketName mybucket -Key $file.filename}
- 25. AWS S3 Multipart Upload Corporate Data center Amazon S3 Bucket Storage Servers Files File systems
- 26. COPY Files from S3 to S3
- 27. Example: Source Bucket : distcptest Region: Tokyo Size : 9GB Files : 90 Target Bucket : distcptestsydney Region: Sydney How to copy objects from S3 bucket to another S3 bucket? Copy files from S3 to S3 with AWSCLI
- 28. $aws s3 sync s3://distcptest/moveiclip s3://distcptestsydney/movieclip * Time: 14 mins Copy files from S3 to S3 with AWSCLI
- 29. Distrubuted Copy using Elastic MapReduce Hadoop cluster Copy files from S3 to S3 with EMR
- 30. 1. Launch EMR cluster with selecting instance size and the number of instances. Copy files from S3 to S3 with EMR $ aws emr add-steps --cluster-id j-XXXXXXXXXX --steps Type=CUSTOM_JAR,Name="S3DistCp step",Jar= /home/hadoop/lib/emr-s3distcp-1.0.jar, Args=["--src,s3://distcptest/moveiclip ","--dest,s3://distcptestsydney/movieclip”] 2. Submit s3DistCp Job to the cluster. (3DistCp is an extension of DistCp that is optimized to work with AWS)
- 31. EMR Job result log Copy files from S3 to S3 with EMR * Time: 3.5 mins (CLI: 14 mins) Bigger cluster, Save more time to copy
- 32. Resizing cluster to more 6 core nodes and test copying Copy files from S3 to S3 with EMR * Time: 1.5 mins
- 33. Design Options on AWS Multi-AZ Cross Region Hybrid IT Single AZ Databases Files and Objects AZ AZ AZ Region Region Compute
- 34. Database Migration – RDS § RDS supports cross region DB snapshot copy § Full and incremental copy § Simultaneously multiple regional copy supported Primary Secondary Secondary
- 35. Database Migration – RDS § A snapshot copied across regions does not include either the parameter group or option group that was used by the DB instance the snapshot was created from.
- 37. $ python copy_db_parameter_group.py --src=src-test -- src_region=ap-northeast-1 --dest=desttestparam -- dest_region=ap-southeast-1 Database Migration – RDS §Using a custom python script, § https://aws.amazon.com/ko/blogs/korea/copy-db-parameters- between-regions-using-aws-cli/ §Python, boto3 library, deepdiff library required
- 38. AWS Database Migration Service
- 39. Purposes of data migration One-time data migration Between on premises and AWS Between Amazon EC2 and Amazon RDS Ongoing Replication Replicate on premises to AWS Replicate AWS to on premises Replicate OLTP to BI Replicate for query offloading
- 40. Ways to migrate data Bulk Load AWS Database Migration Service Oracle Import/Export Oracle Data Pump Network Mode Oracle SQL*Loader Oracle Materialized Views CTAS / INSERT over dblink Ongoing Replication AWS Database Migration Service Oracle Data Pump Network Mode Oracle Materialized Views Oracle GoldenGate
- 41. High-speed database migration prior to AWS DMS EC2 Instance Linux Host On-Premises AWS Availability Zone Oracle DB RDS Oracle Tsunami Tsunami DATA_PUMP_DIR 500GB 175GB ~2.5 hours~2.5 hours Total Time ~7 hours ~3.5 hours ~4 hours
- 43. 10 minutes or less to migration
- 44. Customer Premises Application Users AWS Internet VPN § Start a replication instance § Connect to source and target databases § Select tables, schemas, or databases Let AWS Database Migration Service create tables, load data, and keep them in sync Switch applications over to the target at your convenience Keep your apps running during the migration AWS Database Migration Service
- 45. After migration, use for replication and data integration §Replicate data in on-premises databases to AWS §Replicate OLTP data to Amazon Redshift §Integrate tables from third-party software into your reporting or core OLTP systems §Hybrid cloud is a stepping stone in migration to AWS
- 46. Cost-effective and no upfront costs § T2 pricing starts at $0.018 per Hour for T2.micro § C4 pricing starts at $0.154 per Hour for C4.large § 50GB GP2 storage included with T2 instances § 100GB GP2 storage included with C4 instances § § Data transfer inbound and within AZ is free § Data transfer across AZs starts at $0.01 per GB Swap Logs Cache
- 47. Migrate and replicate between database engines
- 48. Introducing AWS Schema Conversion Tool
- 50. Get help with converting tables, views, and code Schemas Tables Indexes Views Packages Stored Procedures Functions Triggers Sequences User Defined Types Synonyms
- 51. Know exactly where manual edits are needed
- 53. DynamoDB Data Pipeline Region A Region B DynamoDB cross region replication
- 54. DynamoDB cross region replication
- 55. DynamoDB cross region replication
- 56. Design Options on AWS Multi-AZ Cross Region Hybrid IT Single AZ Databases Files and Objects AZ AZ AZ Region Region Compute Network
- 57. Transition traffic with Route53
- 58. Amazon Route 53 weighted routing
- 59. AWS Data Migration Service Continuous Data Replication Passive Amazon Route 53 Database Server Weighted Routing example #1 VM Import/Export AWS Region Database Corporate Data Center 100% Web/ App Server 0%
- 60. AWS Data Migration Service Continuous Data Replication Passive Amazon Route 53 Database Server Weighted Routing example #2 VM Import/Export AWS Region Database Corporate Data Center 90% Web/ App Server 10%
- 61. AWS Data Migration Service Continuous Data Replication Passive Amazon Route 53 Database Server Weighted Routing example #3 VM Import/Export AWS Region Database Corporate Data Center 50% Web/ App Server 50%
- 62. AWS Data Migration Service Continuous Data Replication Passive Amazon Route 53 Database Server Weighted Routing example #4 VM Import/Export AWS Region Database Corporate Data Center 0% Web/ App Server 100%
- 63. Thank you!