Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Gatsby/Contentful to work offline #3977

Closed
wants to merge 4 commits into from

Conversation

uttrasey
Copy link

@uttrasey uttrasey commented Feb 11, 2018

GATSBY_CONTENTFUL_OFFLINE environment variable will make use of a cached fetch results from last successful comms with CMS. Useful if you want to develop/build without an internet connection.

Pretty rudimentary but works.

@gatsbybot
Copy link
Collaborator

Deploy preview for gatsbygram ready!

Built with commit 13fb510

https://deploy-preview-3977--gatsbygram.netlify.com

@uttrasey
Copy link
Author

I'll refactor this and push new logic down into fetch.js where it belongs

@gatsbybot
Copy link
Collaborator

gatsbybot commented Feb 11, 2018

Deploy preview for gatsbygram ready!

Built with commit 97f3b1c

https://deploy-preview-3977--gatsbygram.netlify.com

@@ -29,9 +53,10 @@ module.exports = async ({ spaceId, accessToken, host, syncToken }) => {
console.log(
`Accessing your Contentful space failed. Perhaps you're offline or the spaceId/accessToken is incorrect.`
)
// TODO perhaps continue if there's cached data? That would let
// someone develop a contentful site even if not connected to the internet.
// For prod builds though always fail if we can't get the latest data.
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what the optimal user workflow is here. I've opted for forcing the user to acknowledge a loss of connectivity rather than trying to auto serve cache if network is down.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opt-in require user to know about this feature in first place and if they don't have connectivity they can't actually look for solution for this online.

There are 2 ways to solve it:

  • do automatic fallback and add console message that cached data is used because plugin couldn't connect with contentful api
  • log that plugin can't connect and provide information how to enable this feature
Copy link
Contributor

@pieh pieh Feb 11, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My bad, responded to your comment without reading your code change. I think this is better than other solution as this will cause less confusion that possibly stale data is used and requiring user action means user will acknowledge use of cached data.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool 👍

@pieh
Copy link
Contributor

pieh commented Feb 11, 2018

This is great!

However there is scenario where .cache directory become invalidated (deleted) when you edit your gatsby-node.js or gatsby-config.js and/or do changes to plugins. Perhaps we could add something to prevent deletion of cached source data?

I want to do something like this for wordpress plugin (one of my project has 400 images in media library and everytime I do change in gatsby-node I need redownload them, ugh)

@uttrasey
Copy link
Author

uttrasey commented Feb 11, 2018

@pieh a persistent cache strategy seems like a useful addition, re-pulling 400 images must be annoying and inefficient.

Shall we address that as a separate topic or as part of this PR?

  • My knowledge of the gatsby cache is slim
  • I'd imagine it will need some careful thought / opinion gathering
@pieh
Copy link
Contributor

pieh commented Feb 11, 2018

I think that is out of the scope of this PR as it would involve changes in core, so that would be separate topic. Just figured it's always worth at least mentioning possible gotchas.

@KyleAMathews
Copy link
Contributor

This is really something that needs to be solved in core.

I've been planning to tackle this for v3.

Basically how it'd work is:

  1. make the node & file caches persistent and global
  2. check if online w/ something like https://www.npmjs.com/package/is-online and skip sourcing remote nodes in that case
  3. timeout source plugins that haven't created any nodes after ~5 seconds (e.g. the upstream is down) and ask user if they want to continue with stale data

That being said — this is a pretty non-intrusive PR that seems safe to let in to let you and others start testing this.

Let's just not document it and add a comment in the code saying that this functionality will be removed once core supports offline editing.

Thoughts?

@uttrasey
Copy link
Author

@KyleAMathews yep, makes absolute sense to solve this holistically in core. I'll add the comment and update the PR.


async function writeCacheFile(dir, result) {
try {
await fs.writeJson(`${dir}/${cacheFilename}`, result)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should use path.join.

@@ -63,6 +63,7 @@ exports.sourceNodes = async (
spaceId,
accessToken,
host,
cacheDir: `${store.getState().program.directory}/.cache`,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As should this

@pieh
Copy link
Contributor

pieh commented Jun 7, 2018

@uttrasey It'd be great to get this merged if you have the chance to update it? In order to clean up the PR list I'll close hanging PRs like this over the next few days. But it's always better to merge a PR, so drop a comment here if you'd like to keep this open :) Or feel free to re-open this later.

@Khaledgarbaya
Copy link
Contributor

Hey @pieh @uttrasey let me know if you need help with this to get merged

@pieh
Copy link
Contributor

pieh commented Jun 8, 2018

@Khaledgarbaya I think this just needs to merge master in / resolve conflicts and address Kyle's comments.

Potentially I would also add to console output after Using Contentful Offline cache line that if user tries to edit package.json, gatsby-node.js or gatsby-config.js files in project directory or change any plugin version (I think it would be hard to do when they are offline) - it will invalidate cache and they won't be able to continue using previously cached data.

@pieh
Copy link
Contributor

pieh commented Jun 13, 2018

This is continued by @Khaledgarbaya in #5869 , so I'm closing this as inactive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
5 participants