Issues getting gsutil to use the gcloud activated service account

Question

I am trying to get a service account which I have activated via gcloud auth login to be picked up by gsutil. The big query command line tool bq is picking it up fine.

The order of operations I am following:

First:

gcloud auth activate-service-account --key-file snip.json [email protected]

This results in the output: Activated service account credentials for [[email protected]]

Second: I confirm the credential is active by:

gcloud auth list

Which results in: **Credentialed accounts: - [email protected] (active)...

Third: I see if bq tools picks it up:

bq ls

Which correctly lists the dataset in my project.

Fourth: I try gsutil against my bucket. I have added my service account user as an owner to this bucket to ensure it has permissions.

gsutil ls gs://snip

This results in: a long wait, then eventually: You are attempting to access protected data with no configured credentials. Please visit https://cloud.google.com/console#/project and sign up for an account, and then run the "gcloud auth login" command to configure gsutil to use these credentials.

Then I use the debug command:

gsutil -D

The output:

gsutil version: 4.11
checksum: snip (OK)
boto version: 2.30.0
python version: 2.7.5 (default, Jun 17 2014, 18:11:42) [GCC 4.8.2 20140120 (Red Hat 4.8.2-16)]
OS: Linux 3.10.0-123.20.1.el7.x86_64
multiprocessing available: True
using cloud sdk: True
config path: no config found
gsutil path: /home/blah/google-cloud-sdk/platform/gsutil/gsutil
compiled crcmod: False
installed via package manager: False
editable install: False
Command being run: /home/blah/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=snip -D
config_file_list: []
config: [('debug', '0'), ('working_dir', '/mnt/pyami'),'https_validate_certificates', 'true'), ('debug', '0'), ('working_dir', '/mnt/pyami'), ('default_project_id', 'snip')]

After this, if I add my own user id to the credential store via gcloud auth login, gsutil ls gs://snip works fine. If I then switch the service account to be active instead of my user id (using gcloud config set account) it then doesn't work again - I get the same error message as above.

Last piece of info - I am running this on my own VM, not within GCE.

Answer 1

I was able to reproduce the issue. It seems to be happening only with JSON key files. Google is aware of this issue as per this. Until it's resolved I'd recommend either authenticating using a P12 key file, or to add the default project service account as an owner on the bucket you're trying to access.

Answer 2

You need to authorize the service account on the bucket from the cloud console click the "Edit bucket permission" and then add a permission for the email address of the service account.

Answer 3

Admittedly I am not sure this is the answer to the question (please let me know and I will delete it), but seeing that there is no accepted answer for this and the two answers I see didn't solve the issue for me, here is what fixed the gsutil issue for me:

I re-initialized gcloud on the project level via the console shell (click the >_ icon in the top right of the console to get the shell), then run something like this: user@direct-topic-1234:~$ gcloud init

It will ask you a bunch of questions. I think choosing "Re-initialize this configuration [cloudshell-[#]] with new settings" was key, but I am not 100% sure.

After I finished that gsutil started to work. I was doing something different, but I am hoping this will help with this question. I was doing:

gsutil mv gs://bucketname/filename instance-name:/a/path/filename

(In the above, I am moving filename file from a project bucket to an instance instance-name and directory /a/path/.)

Mid-way down this link there is some helpful info: https://cloud.google.com/compute/docs/disks/gcs-buckets

This fixed gsutil for me! (I think it worked because before doing the above, my gsutil was configured for a zone that my bucket and instance were not in.)