Timeline for Exporting data from Google Cloud Storage to Amazon S3
Current License: CC BY-SA 3.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Dec 4, 2023 at 13:14 | comment | added | Mario Souza |
I was getting 403. gcloud did the trick: gcloud storage rsync gs://{SOURCE} s3://{DESTINATION} --recursive
|
|
| Dec 4, 2021 at 12:42 | comment | added | Rishabh Rusia | @MikeSchwartz Will this work if I need to transfer files to AWS Mainland China, Beijing region S3 bucket ?. I found that they allow S3 non-China to China region transfer (as in link below), but not sure on cross cloud aws.amazon.com/blogs/storage/… | |
| Dec 12, 2019 at 16:30 | comment | added | Mike Schwartz | raghav@ - you could shard the gsutil rsync, running on several VMs, with each handling a subset of the copying - see stackoverflow.com/questions/31492872/… for an example of sharding (in that case it's listing, but hopefully that gives you an idea of how to do it for copying too). | |
| May 30, 2019 at 10:05 | comment | added | raghav | I'm running the above command on a google vm instance where download/upload speed is ~ 500-600 mbps, and the data to be migrated is 400gb. The process is taking very long. Is there any way I can speed up the migration? | |
| Feb 19, 2019 at 17:52 | comment | added | Pathead | This works but unfortunately gsutil does not support multipart uploads, which the S3 API requires for files larger than 5GB. | |
| Jan 31, 2019 at 16:19 | comment | added | Mike Schwartz | The boto config fiile is used for credentials if you installed standalone gsutil, while the credential store is used if you installed gsutil as part of the Google Cloud SDK (cloud.google.com/storage/docs/gsutil_install#sdk-install) | |
| Apr 5, 2018 at 10:30 | comment | added | MJK | Before that you need to add your aws credentials in boto.cfg file | |
| Jan 2, 2018 at 7:05 | comment | added | Nirojan Selvanathan | Im getting an error for the same operation although the s3 bucket has public read and write access. Hope I'm not missing anything here. The gsutil was executed inside the google cloud shell. Error Message - ERROR 1228 14:00:22.190043 utils.py] Unable to read instance data, giving up Failure: No handler was ready to authenticate. 4 handlers were checked. ['HmacAuthV1Handler', 'DevshellAuth', 'OAuth2Auth', 'OAuth2ServiceAccountAuth'] Check your credentials. | |
| Sep 6, 2016 at 16:03 | history | edited | Mike Schwartz | CC BY-SA 3.0 |
added 36 characters in body
|
| Sep 5, 2016 at 23:34 | vote | accept | Onca | ||
| Sep 5, 2016 at 14:59 | history | answered | Mike Schwartz | CC BY-SA 3.0 |