How to list all files in Google Storage bucket in a short time?

Question

I have a Google Storage bucket that contains more than 20k+ filenames. Is there any way to list all the filenames in the bucket in a short time?

Answer 1

It depends on what you mean by "short", but:

One thing you can do to speed up listing a bucket is to shard the listing operation. For example, if your bucket has objects that begin with English alphabetic characters you could list each letter in parallel and combine the results. You could do this with gsutil in bash like this:

gsutil ls gs://your-bucket/a* > a.out &
gsutil ls gs://your-bucket/b* > b.out &
...
gsutil ls gs://your-bucket/b* > z.out &
wait
cat ?.out > listing.out

If your bucket has objects with different naming you'd have to adjust how you do the sharding.