Using gsutil to rsync from GCP to S3

Question

There's a similar question with answers here. However, the top answer there does not work for me:

If I use:

gsutil -m rsync -r gs://your-gcs-bucket s3://your-s3-bucket

It fails for me with:

The authorization header is malformed; the region 'us-east-1' is wrong; expecting 'eu-west-2'

Ok, rclone does work but, still, why not with gsutil rync?

I'm in London region by the way.

One possible clue might be this article in AWS which describes some reasons and fixes ... does any of this sound applicable to your story? docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/… — Kolban, Commented May 25, 2020 at 17:19
I'm familiar to that and in almost all commands involved in GCP or AWS there is an option for --region or --zone. In this particular case, which is more complicated, I want to set the region for AWS within a call from a GCP command. I've tried using ENV set up or default setup with GCP and AWS but for no avail so far. I wish I could understand hor rclone is sorting this issue. — alanwilter, Commented May 25, 2020 at 20:28
Just a wild guess ... maybe edit ~/.boto I'm sesnsing a couple of possibilities ... either host=s3.eu-west-2.amazonaws.com under an s3 section or https_validate_certificates = false under the Boto section. — Kolban, Commented May 25, 2020 at 21:29
Thanks for your tip @Kolban, indeed, editing ~/.boto and adding under [Credentials] s3_host = s3.eu-west-2.amazonaws.com did solve the issue. — alanwilter, Commented May 25, 2020 at 23:12

SapphireSun · Accepted Answer · 2021-05-06 19:22:46Z

5

Associated with the gsutil command is a configuration file called ~/.boto. Within this file one can add/code:

[Credentials]
s3_host=s3.eu-west-2.amazonaws.com

For details on this file and related information, please see:

SapphireSun

9,34811 gold badges48 silver badges59 bronze badges

answered May 26, 2020 at 4:05

Kolban

15k3 gold badges40 silver badges70 bronze badges

Add a comment |

1 Answer 1