What makes me to migrate entire VPC JAWS PANKRATION 2021
•
0 likes•692 views
2021/11/21 JAWS PANKRATION 2021で発表した時の資料です。 日本語版の資料は以下です。 https://speakerdeck.com/coopsapporo/what-makes-me-to-migrate-entire-vpc
Report
Share
What makes me to migrate entire VPC JAWS PANKRATION 2021
- 1. What makes me to migrate entire VPC JAWS-UG Architecture / JAWS-UG Systems Administrators Naomi Yamasaki
- 2. AWS SAMURAI 2015 JAWS-UG Architecture JAWS-UG System Administrators Naomi Yamasaki Infrastructure Engineer Digital Headquarters Consumers CO-OP Sapporo @nao_spon I ♡ Route53 IAM Organizations
- 3. Why do I migrate entire VPC?
- 4. We are going to AWS All in!! ● 190 Systems, 650 Servers at Datacenter ● Migrate with CloudEndure Migration ● Think CloudFirst for new systems
- 5. I love AWS Organizations ● Control accounts, service and user role ○ Control Tower ○ SCP ○ OU ○ AWS SSO ● Standardize environments and Policy ○ CloudFormation Stack Sets ○ AWS Security Hub ○ AWS Config
- 6. Account designing ● Basically, 3 accounts for 1 system ○ Separate accounts for development, staging, production ○ Some of them have fewer accounts for some reason ● Network ○ Each VPCs are connected with Transit Gateway ○ AWS and Datacenter is connected with Direct Connect
- 7. Increasing AWS accounts Migrated systems accounts: 59 New systems accounts: 67 Other accounts: 46
- 8. Why do I migrate entire VPC?
- 9. There are 3 accounts created before 2020 ● They created before we started using AWS in earnest ● There was intended to be a small start at the time ● We want to put these accounts in Organizations too.
- 10. Why we include those accounts in AWS Organizations Not under the control of Organizations means configure these settings individual ○ Control accounts, service and user role ■ Control Tower ■ SCP ■ OU ■ AWS SSO ○ Standardize environments and Policy ■ CloudFormation Stack Sets ■ AWS Security Hub ■ AWS Config
- 11. Most effect and wanted things
- 12. Most effect and wanted things
- 13. Problems in attaching to Transit Gateway
- 14. Problems in attaching to Transit Gateway ● Duplicated network segment ● Too much big CIDR blocks for VPC
- 15. We decided to migrate VPC to another account ● Prefer ○ Separate accounts for each environments ○ Separate frontend and backend system’s resources being together ○ Less confusing than changing the current environment. ● Not prefer ○ Minimize the VPCs CIDR blocks ○ Change current go live production environment
- 16. Before
- 17. After
- 18. Migration schedule 1st: Staging environment 2nd: Production environment 3rd: Develop environment
- 19. The most difficult part of the migration process was... Aurora migration
- 20. Which method to choose for DB migration? ● Prefer simple and easy way ● Make it easy to create a replication environment ● Binary log replication seemed to be a pain to configure ● DMS is looks like easy way
- 21. Which method to choose for DB migration? ● Prefer simple and easy way ● Make it easy to create a replication environment ● Binary log replication seemed to be a pain to configure ● DMS is looks like easy way♡
- 22. Not easy way...
- 23. Traps in DMS ● Defaults, unique keys, comments were disappear and the number of int digits had changed ● Replication was failed on Out Of Memory in nighttime batch.
- 24. Nighttime batch with DMS to be a nightmare batch ● After the nightmare batch, failed replication is replicated little by little ● I tried scaled up maximum the DMS replication instance, but nightmare batch made me nightmare… ● Out Of Memory with dms.r5.24xlarge: 96vCPUs, 768GiB memory
- 25. We gave up on using DMS ● 1 week left to the staging environment switchover ● 3 weeks left to production environment switchover ● We don't want to take a different approach to switchover each staging and production environment ● We had no time to try Binary log replication.
- 26. I couldn't say “Let’s do it!” anymore...
- 27. How can we resolve this issue? ● The service is stopped at 2:00 to 4:00 AM due to nighttime batch, so we can take down time at that time. ● We have a time to shift for the nighttime batches. ● We decided to sharing DB snapshots and restore from ● It was the best way for us at that time
- 28. Another issue with AWS KMS key DB snapshot cannot be shared to another account when using default aws/rds KMS key to encrypt the database
- 29. How to resolve it? ● Work at the current account ○ Create Customer managed key on KMS ○ Put permissions to IAM User or Role on Key Policy ○ Create DB snapshot ○ Copy DB snapshot and choose the KMS Key ○ Share the copied DB snapshot to new account ● Work at the new account ○ Copy the shared DB snapshot and choose KMS Key as ‘aws/rds’ ○ Create Aurora cluster from copied DB snapshot
- 30. CloudFormation issues ● There were manually created resources ● Some resources were created with CloudFormation, but some of them manually modified after all
- 31. CloudFormation issues ● I want to use the same templates for each staging, production and development environments ● I had to rewrite the all of the templates for import resources into CloudFormation stack, so I re-creating everything to new.
- 32. CloudFormation vs CDK ● Need low context description when I try to accurately resource settings in detail. ● I'm more familiar with YAML.
- 33. ● Using describe commands by AWS CLI ● Mappings like a variable How did I create CloudFormation templates
- 34. ● 1 templates for 1 resource ● I created: ○ Staging environment: 44 templates ○ Production environment: 60 templates ● Just copy templates and change the Mappings value How many CloudFormation templates I created
- 35. We’ve done it!! ● There was no big trouble. ● Also no big problems in service delivery after switched ● Development environment migration will be postponed until the major upcoming release have done
- 36. What I learned... ● Ensure good network design. ● Don't create a VPC with too large CIDR block ● DMS will match if there is not a large amount of processing at once ● Understand what your database doing ● Choose the best migration method as my DB ● Infrastructure as Code is soooooooooooo important!
- 37. The best way is sometimes different from the best practice
- 38. 2 another accounts are left!! Again?!