feat(contentful): add retry logic to asset downloading

[axe312ger]

Contentful users with a huge number of assets might run into issues when requesting to many low quality image previews.

This adds a concurrency of 100 to the download process and a retry logic based on https://github.com/JustinBeckwith/retry-axios.

The very same logic will be applied when detecting the dominant color and creating a traced svg variant, for both gatsby-image and gatsby-plugin-image

[wardpeet]

@axe312gr could you look at https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-source-filesystem/src/create-remote-file-node.js and move to got instead? Got has everything built in. For mocking I would suggest https://mswjs.io/ which I also used in https://github.com/gatsbyjs/gatsby/blob/master/packages/gatsby-source-filesystem/src/__tests__/create-remote-file-node-integration.js.

I'm fine with using nock for now but I would like you to move to got instead. The plan is to move to one fetching library that we can fine tune to make sure it re-uses connections, ...

Nvm, we already using axios

[wardpeet]

Added some comments. I'm curious if it would be possible to have base64 part of contentful sdk? This way we can get the base64 strings in sourceNodes and properly cache things.

Does axios-retry also take care of incomplete downloads? We've seen in source-filesystem that status ok is given but the file did not download all bytes

[axe312ger]

I'm curious if it would be possible to have base64 part of contentful sdk?

No, this is out of scope of our SDK. Ours SDK purpose is to simplify communication with our API, creating base64 variants is not a feature of our API therefore should not be part of the SDK,

[wardpeet]

Let's see if tests pass. Code looks good! Thank you!

[njbmartin]

This slipped past review, reporter never gets passed in from anywhere, so the retry logic never actually works. Test worked due to reporter being stubbed. See #31590

[axe312ger]

@njbmartin yes, you are right :(

gatsby/packages/gatsby-source-contentful/src/cache-image.js

Line 62 in 0d832d9

downloadWithRetry({

gatsby/packages/gatsby-source-contentful/src/extend-node-type.js

Line 114 in 71ec0cd

const imageResponse = await downloadWithRetry({

[axe312ger]

@njbmartin Thanks again, opened a PR to fix this in the next version: #31608