AWS CLI - using credentials unable to list buckets, using profile I can list buckets

Question

I'm trying to understand a bit the following situation. I have set $HOME/.aws/credentials as follows

[aws-users-andres]
aws_access_key_id = accesskey
aws_secret_access_key = UJPYNsecretkey

this credentials are refreshed periodically using some scripts and MFA. $AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN are set in the environment.

Then I also have $HOME/.aws/config file with a set of profiles that my company is using to access different resources:

[profile profile_name]
role_arn = arn:aws:iam::1464549XXXX:role/profile_name
credential_source = Environment

Now if I try to list the buckets:

aws s3 ls

The output is "An error occurred (AccessDenied) when calling the ListBuckets operation: Access Denied"

But running using the profile

aws s3 ls --profile profile_name 

I'm able to list the buckets in our S3.

I'd like to list the buckets without specifying the profile or set the default behavior to use profile_name . I was recently tasked with a task to periodically sync some google cloud storage bucket with s3 resources, and was following the ideas from here Exporting data from Google Cloud Storage to Amazon S3 but the gsutil command fails ( Forbidden access to S3 )

I have tried to use AWS_PROFILE and AWS_DEFAULT_PROFILE env var set as profile_name but that doesn't seem to work.

I appreciate any help and guidance. I could write some ad hoc script to copy and upload the needed files from gcloud to s3 but I was hoping gsutil would be a better option.

Thanks in advance.

Answer 1

Using inline environment variables:

AWS_ACCESS_KEY_ID="THISISMYKEY" \
AWS_SECRET_ACCESS_KEY="THIS_IS_SECRET_KEY" \
aws s3 ls 

is the quickest but least secure way of doing this. It's insecure because running history | grep "AWS_ACCESS_KEY_ID" easily exposes the credentials used.

A better approach is to not use inline shell variables, avoid setting the credentials manually on those places and just utilise the built-in AWS CLI command for configuring profiles for your machine.

$ aws configure --profile allan
$ AWS Access Key ID [None]: THISISMYKEY
$ AWS Secret Access Key [None]: THIS_IS_SECRET_KEY
$ Default region name [None]: ap-southeast-1
$ Default output format [None]:

aws s3 ls --profile allan 

If these two options don't work, consider seeking help from the person that issued your AWS credentials. The credentials may be stale, incorrect or lack permissions to query S3 buckets.