Does anyone have experience with Google Cloud Storage? Is it made for a potential 10K of parallel requests to download 10-70mb files? Or would that require a CDN?
2 Answers
https://cloud.google.com/storage/quotas
There is no limit to reads of an object. Buckets initially support roughly 5000 reads per second and then scale as needed.
You may have trouble if you frequently go from zero to 10k parallel requests (assuming they're really parallel, and not, say, "in the same couple of minutes") rapidly:
As a bucket approaches its IO capacity limit, Cloud Storage typically takes on the order of minutes to detect and accordingly redistribute the load across more servers. Consequently, if the request rate on your bucket increases faster than Cloud Storage can perform this redistribution, you may run into temporary limits, specifically higher latency and error rates. Ramping up the request rate gradually for your buckets, as described below, avoids such latency and errors.
-
Thanks! But I have a little question on this: It sounds like this limit is per bucket... so even if only 5-10% of the files in the bucket reach that limit (like potentially in my case), the whole bucket has to be redistributed?– Dave A.Commented Jul 12, 2018 at 16:57
-
@DaveA. Are you expecting to go from 0/second to 10,000/second in a short period of time? If you're going to see consistent high loads, Google should adjust. I'd presume if you've got a million requests a second for one object they'll duplicate it across multiple GCS backend servers.– ceejayozCommented Jul 12, 2018 at 17:14
Yes, GCS can handle 10k parallel requests to download files.
-
This is not 100% true (but OP's definition of "parallel" probably doesn't mean "in the same second with no warm-up", so your answer is likely true in their specific case). See my answer.– ceejayozCommented Jul 10, 2018 at 18:52
-
1Good point about instant spin-up of that many parallel requests. Thanks for adding the doc excerpts about that. Commented Jul 11, 2018 at 16:00