0

I was trying to save two files to GCP Storage using the following commands in a Jupyter Notebook:

!gsutil cp ./dist/my_custom_code-0.1.tar.gz gs://$BUCKET_NAME/custom_prediction_routine_tutorial/my_custom_code-0.1.tar.gz
!gsutil cp model.h5 preprocessor.pkl gs://$BUCKET_NAME/custom_prediction_routine_tutorial/model/

The bucket has been created properly since I can see it in the bucket list on GCP. Also in Permissions for the bucket, I can see the service account created. Plus, I made sure the environment variable is set by running:

export GOOGLE_APPLICATION_CREDENTIALS="/home/george/Documents/Credentials/prediction-routine-new-b7a445077e61.json"

This can be verified by running this in Python:

import os

print('Credendtials from environ: {}'.format(os.environ.get('GOOGLE_APPLICATION_CREDENTIALS')))

which shows:

Credentials from environ: /home/george/Documents/Credentials/prediction-routine-new-b7a445077e61.json

And I do have the json file stored at the specified location. However, when I tried to save files using the commands shown at the top, I kept getting this error message:

AccessDeniedException: 403 george***@gmail.com does not have storage.objects.list access to the Google Cloud Storage bucket.
Copying file://model.h5 [Content-Type=application/octet-stream]...
AccessDeniedException: 403 george***@gmail.com does not have storage.objects.create access to the Google Cloud Storage object.

So the question is, how come Google Storage is not using my service account and keeps using my user account?

UPDATE

After activating the service account for the project as pointed out by @Hao Z, GCP is using my service account now. However, I do have the permissions set for this service account...

IAM Permissions

UPDATE 2

This seems to be a known issue: https://github.com/GoogleCloudPlatform/gsutil/issues/546

Improve this question
3
  • 1
    Do you authenticate the gcloud tool with your account via run command? gcloud auth activate-service-account \ > --key-file=${GOOGLE_APPLICATION_CREDENTIALS}
    – Hao Z
    Commented Jun 15, 2020 at 23:34
  • Thank you @HaoZ! After doing this, now it's using my service account since it's saying "AccessDeniedException: 403 prediction-routine-new@prediction-routine-test.iam.gserviceaccount.com does not have storage.objects.list access to the Google Cloud Storage bucket." However, under IAM permissions, it does say there is a Storage Object Admin role... as shown in the image in my post above. Would you mind providing some further pointers?
    – George Liu
    Commented Jun 16, 2020 at 3:17
  • 1
    You can re-grant the Storage Object Admin role to your account prediction-routine-new@prediction-routine-test.iam.gserviceaccount.com, or grant the Storage Object Create and Storage Object Viewer roles to this account. I suggest you use Policy Troubleshooter to examine all Cloud IAM policies that apply to the resource.
    – Hao Z
    Commented Jun 17, 2020 at 0:07

2 Answers 2

Reset to default
1

Check How to use Service Accounts with gsutil, for uploading to CS + BigQuery

Relevant bit:

Download service account key file, and put it in e.g. /etc/backup-account.json
gcloud auth activate-service-account --key-file /etc/backup-account.json

Or you can do gsutil -i to impersonate a service account. Use 'gsutil help creds' for more info. I guess the env variable is just used by the Python SDK and not by the CLI.

Improve this answer
0

I was able to resolve this in the following steps:

First, Using the way suggested by @Hao Z above, I was able to activate the service account in Jupyter Notebook using:

!gcloud auth activate-service-account \
  prediction-routine-new@prediction-routine-test.iam.gserviceaccount.com \
          --key-file=/home/george/Documents/Credentials/prediction-routine-new-b7a445077e61.json \
          --project=prediction-routine-test

Second, I changed the bucket name used after realizing that I was using the wrong name - it should be "prediction-routine" instead of "prediction-routine-bucket".

BUCKET_NAME="prediction-routine"

Third, I changed the role from "Storage Object Admmin" to "Storage Admin" for the service account's permissions.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.