3

What is the best way to authenticate to Google Cloud Storage Bucket from a shell script (To be scheduled to run daily/hourly) using a service account? I have gone through the below link, but I still have some doubts regarding the login process.

How to use Service Accounts with gsutil, for uploading to CS + BigQuery

  1. Are the below mentioned login steps a one-time process? If yes how does the login work for subsequent executions?

    My understanding is that the below commands writes content to the .boto file which is used in subsequent executions? But according to below link - it writes to a separate json file inside .config/gcloud? Does gsutil support creating boto files with service account info?

    In such a case what is the use of a .boto file ? and why/when do we need to pass it via BOTO_PATH/BOTO_CONFIG?

    In gsutil (standalone), login using below steps

    gsutil config -e

    Optionally -o to output to a file other than ~/.boto

    gsutil as part of gcloud

    gcloud auth activate-service-account [email protected] --key-file=/path/key.json --project=PROJECT_ID

  2. What is the best way to prevent intervention from other scripts?

    For example, let us assume we have shell script S1, connecting to project P1 to upload data to Bucket B1, If another shell script say S2 is triggered at exactly the same time connecting to Project P2 uploading to Bucket B2, will it cause an issue? What is the best practice to avoid such issues?

  3. Is it possible to limit the login to only the time of script execution? Say, the script is scheduled using cron to run at 10:00 AM UTC and the script completes its execution by 10:30 AM UTC. Is it possible to prevent any actions in the time between 10:30 till next run? In other words is it possible to log out and then login programatically without intervention?

Environment: Centos

Improve this question

1 Answer 1

Reset to default
0

The principle of BOTO file is exactly to answer your question 2. You can have 2 credentials that have access to 2 different buckets. Create 2 boto file and use the correct one for each script.

For the 3rd question it's possible to set condition on the bucket access.

Select a bucket and go to right-hand side in the info panel, and click on add credential. enter image description here

Then, add your credential, your role, and click on add condition (you must set the uniform permission definition on the bucket to have available that feature) enter image description here

And then define a condition to allow the permission after 10am your timezone and before 11am your timezone (you don't have minute granularity) enter image description here

Improve this answer
Recognized by Google Cloud Collective
5
  • Thank you very much for your response. 1. Regarding #1, is my understanding that both config and auth create boto files ? 2. Regarding #2, Is it possible to pass a specific boto_file for authentication at runtime ? 3. Regarding #3, we cannot be sure that the script would complete in one hour ? say we have a network issue and we need to restart after 11 AM, this setting would not allow for it right ?
    – rainingdistros
    Commented May 1, 2022 at 12:37
  • #1 Boto file is only created by gsutil config. The gcloud auth works only for gcloud, gsutil and bq. No Boto file create, but credential files atwell-known location (see ADC for more detail). #2 Use inline command BOTO_PATH=/centralhub/boto.cfg:home/jane/.boto gsutil cp .... More details here #3 Yes, it's a hard blocker manage by IAM. You have to plan an icebreaker if you have to run job outside the planned time window.
    – guillaume blaquiere
    Commented May 1, 2022 at 18:56
  • Hi @guillaume blaquiere, Thank you so much for your response. #1. The info on ADC was very helpful. Can I assume that provided I have the Service Account Key File - I need not run gsutil config -e or gcloud auth I could just set export GOOGLE_APPLICATION_CREDENTIALS="KEY_PATH" at the beginning of the shell script and proceed with the gsutil commands ? This would also solve #3 right since login is only applicable for current session ? #2. As a best practice, which is preferred, login via BOTO files or ADC ?
    – rainingdistros
    Commented May 2, 2022 at 7:30
  • BOTO is a more standard/compliant solution if you want to use also AWS S3. You can also add additional config parameter in the BOTO. You can also directly with gsutil, but you have to update all your command, in the BOTO file you centralize all the configs. In term of security, it's the same. And yes, you can use GOOGLE_APPLICATION_CREDENTIALS="KEY_PATH" gsutil .... to run a command. Don't export the value because if you have concurrent script you can override the value for others.
    – guillaume blaquiere
    Commented May 2, 2022 at 12:13
  • Wonderful...thank you very much - I will try and let you know for any issues... Thank you very much....
    – rainingdistros
    Commented May 2, 2022 at 14:24

Not the answer you're looking for? Browse other questions tagged or ask your own question.