0

Trying to pull the data from AWS S3 to Google Cloud Storage by using Storage Transfer.

I have S3 bucket "Access key ID" and "Secret access key" and able to copy from gsutil (from laptop) but it is throwing permission error from Storage transfer.

Not understand how it is working with gsutil but not with storage transfer.

Error Message:

Invalid access key. Make sure the access key for your S3 bucket is correct, or set the bucket permissions to Grant Everyone.

Is there any way to use gsutil in compute engine to copy files from S3 to GCS?

Thanks,

Improve this question
4
  • Storage transfer should work. Have you considered enabling S3 bucket logs and CloudTrail to see if any of the requests are arriving? And of course verified that your access key and secret have no copy/paste errors?
    – Michael - sqlbot
    Commented Jun 27, 2017 at 1:28
  • I am not the owner of the S3 bucket to enable S3 bucket logs. I checked with S3 bucket owner and he confirmed that S3 bucket have proper permissions and I am able to copy with gsutil by adding key and secret in boto file. Key and Secret are correct, I cross checked.
    – user374374
    Commented Jun 27, 2017 at 1:36
  • There may be a permission that storage transfer needs that gsutil doesn't need, such as s3:GetObjectAcl. (That's a guess.) Without access to the user and bucket policy, it's tricky to troubleshoot. But, yes, you can use gsutil in Google Compute Engine, or in EC2, or anywhere you have a server with bandwidth. It won't make any difference where you run it, as long as it is geographically close to both buckets.
    – Michael - sqlbot
    Commented Jun 27, 2017 at 1:40
  • Thanks Michael. In compute engine where I need to pass S3 bucket Key and Secrete information, I couldn't find boto file. When I tested in windows laptop I entered key and secrete key information in boto file and tested.
    – user374374
    Commented Jun 27, 2017 at 2:09

1 Answer 1

Reset to default
1

Created .boto file in home directory with the following lines. gsutil is able to connect to aws S3 bucket. 

[Credentials]
aws_access_key_id = <key>
aws_secret_access_key = <secret key>

Thanks,

Improve this answer
1
  • My company has access id, key and session token. This doesn't work for me. I got <Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>GE3PP3K1YSY3ET4C</RequestId><HostId>7s02FbN36n8Vxvevzz1/9tR8q+nu/wsDsnsu+ThMhB7qUOMcTZhwoHEVq3FilsoAXplNnIo2z0M=</HostId></Error>
    – saawgr
    Commented Apr 15 at 5:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.