[WIP] Replace global data.json with page-data.json per page #13004

Moocar · 2019-04-01T21:20:42Z

Description

Removes the global data.json file and instead provides a page-data.json for each page on a gatsby site. See #11982 for background.

This PR is a draft and should not be merged. I'll be providing a series of smaller PRs instead. Reviewers can always reference this PR to see the full context of the changes. That said, if you're reading this PR and spot any problems, then please do leave some feedback!

Benefits/Trade offs

pros

no more downloading data.json to the browser. For 10'000 page sites, this file was already approaching 500kb compressed.
navigating to prefetched pages should be faster since there's no data.json to search through
production-app.js webpack build no longer references pages or their data. Which will make incremental builds much easier to implement in the future

cons

Requesting a page from the server now requires 2 requests. First page-data.json, then the component. Though prefetched pages still do this in the background.
Since page data isn't immutable anymore, we can't rely on infinite browser caching
Some potential breaking changes in loader.getResourceURLsForPathname

Sup-PRs

Breaking up these PRs is hard. To make it a bit easier, I'm going to have gatsby saving two sources of data to disk. data.json and static/d/..., as well as the new page-data.json. This will require duplication of memory and disk, so we won't be able to merge to master. But we will be able to gradually introduce new functionality, and once it's ready, delete the old data sources and merge to master. I'm not a fan of long lived non-master branches, but I think it makes sense here.

The long lived branch is per-page-manifest. The PRs that will merge directly into that are:

TODO

Bad handling of timeouts etc https://sentry.io/organizations/gatsby-inc-9z/issues/1054501319/?project=1471074&statsPeriod=24h
page-data.json requested a LOT (resulting in 304s)
- From my reading, it looks like these are all coming from sentry's breadcrumbs.js file.
use networkFirst for workbox for all json files not in /static. Reason: page-data.json is currently staleWhileRevalidate, which doesn't make sense since it's dynamic. See offline-plugin/gatsby-node.js. Double check if networkFIrst is correct, given we're on Workbox 3.
include more information in loader.js errors. E.g whole request object

TODO but won't block release

Expire old page data from production-app #13430
Review "behind the scenes" docs. They'll be very out of date now.
public/static/d/.... isn't used except for static query results. So we should ask the user to delete their public directory when they run gatsby with this code for the first time? It's not required.
Add prefetch for linked component modules to page HTML

Already reviewed and Merged:

Merged against per-page-manifest

Related Issues

cleanup bootstrap moved page-query-runner to index use finishBootstrap() fix query-runner type annotation refactor query-runner/index doc sections queryjobs refactor make query refactor enqueueQueryID -> enqueueExtractedQueryId

Also moved pages-writer.js and redirects-writer.js to src/bootstrap

me4502 · 2019-05-24T05:50:47Z

It was on 96 for performance, but the main win is in the Performance metrics

These are now all ticks, and the diagnostics section is empty.

KyleAMathews · 2019-05-24T06:48:42Z

Nice! 📣 Did it make a difference on build times at all?

…

On Thu, May 23, 2019, 10:50 PM Matthew Miller ***@***.***> wrote: It was on 96 for performance, but the main win is in the Performance metrics [image: image] <https://user-images.githubusercontent.com/546754/58305503-abae2e00-7e3b-11e9-9cff-f8085baa58d6.png> These are now all ticks, and the diagnostics section is empty. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#13004?email_source=notifications&email_token=AAARLBYJWF36BK4RAGKRZODPW563TA5CNFSM4HC2VINKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWEHCOA#issuecomment-495481144>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAARLB5VZN2SDQ7IVXFNXNLPW563TANCNFSM4HC2VINA> .

me4502 · 2019-05-24T06:51:25Z

Actually yeah, 15 minutes down to 9. And I except it to have massively lowered deploy times due to less changed files. I haven't ran multiple deploys yet to test though.

KyleAMathews · 2019-05-27T18:12:49Z

@me4502 oh wow! That's kinda surprising actually — where'd the 6 minutes get saved?

**Note: merges to `per-page-manifest`, not master. See gatsbyjs#13004 for more info** ## Description This PR saves the `page-data.json` during query running. In order to minimize the size of PRs, my strategy is to save the page-data.json along with the normal query results, and then gradually shift functionality over to use `page-data.json` instead of `data.json`. Once all those PRs are merged, we'll be able to go back and delete the static query results, jsonNames, dataPaths, data.json etc. It does mean that we'll be storing double the amount of query results on disk. Hopefully that's ok in the interim. Compilation-hash will be added in future PRs. ## Related Issues - Sub-PR of gatsbyjs#13004

Moocar · 2019-05-27T21:44:34Z

OK, we're ready to finally merge per-page-manifest into master. I just have one concern before we do. I've changed an official browser API onPostPrefetchPathname -> onPostPrefetch. It's only used by the gatsby-plugin-offline plugin (that we know of). So if we merge per-page-manifest to master, publish, and then a site upgrades gatsby but not gatsby-plugin-offline, they'll get an error:

Your plugins must export known APIs from their gatsby-browser.js.
The following exports aren't APIs. Perhaps you made a typo or your plugin is outdated?

See https://www.gatsbyjs.org/docs/browser-apis/ for the list of Gatsby Browser APIs

- The plugin \"gatsby-plugin-offline@2.1.1\" is exporting a variable named \"onPostPrefetchPathname\" which isn't an API.

Perhaps you meant to export \"onPrefetchPathname\"?
error Command failed with exit code 1

Basically it's a breaking change. And we need users to update both gatsby and gatsby-plugin-offline at the same time. Which to my understanding isn't possible (but happy to be shown otherwise). I'll look at how much work it will be to make it backwards compatible.

me4502 · 2019-05-27T23:56:23Z

@KyleAMathews Seems to just be generally all over the build:
Old:

[2019-05-20T03:34:23.683Z] success open and validate gatsby-configs — 0.015 s
[2019-05-20T03:34:23.944Z] success load plugins — 4.015 s
[2019-05-20T03:34:23.944Z] success onPreInit — 0.006 s
[2019-05-20T03:34:23.944Z] success delete html and css files from previous builds — 0.007 s
[2019-05-20T03:34:23.944Z] success initialize cache — 0.009 s
[2019-05-20T03:34:23.944Z] success start nodes db — 0.005 s
[2019-05-20T03:34:24.203Z] success copy gatsby files — 0.042 s
[2019-05-20T03:34:24.203Z] success onPreBootstrap — 0.014 s
[2019-05-20T03:34:24.203Z] Starting to fetch data from Prismic
[2019-05-20T03:34:24.462Z] Fetch Prismic data: 387.842ms
[2019-05-20T03:35:03.976Z] success source and transform nodes — 35.679 s
[2019-05-20T03:35:03.976Z] success building schema — 1.061 s
[2019-05-20T03:38:25.535Z] success createPages — 193.127 s
[2019-05-20T03:38:25.536Z] success createPagesStatefully — 0.075 s
[2019-05-20T03:38:25.536Z] success onPreExtractQueries — 0.027 s
[2019-05-20T03:38:25.536Z] success update schema — 0.000 s
[2019-05-20T03:38:25.536Z] success extract queries from components — 0.389 s
[2019-05-20T03:38:25.536Z] success run static queries — 0.099 s — 3/3 30.93 queries/second
[2019-05-20T03:39:31.986Z] success run page queries — 75.684 s — 55490/55490 736.00 queries/second
[2019-05-20T03:39:33.397Z] success write out page data — 2.811 s
[2019-05-20T03:39:33.397Z] success write out redirect data — 0.001 s
[2019-05-20T03:42:10.295Z] 
[2019-05-20T03:42:10.295Z] success Build manifest and related icons — 0.257 s
[2019-05-20T03:42:10.295Z] success onPostBootstrap — 0.260 s
[2019-05-20T03:42:10.295Z] 
[2019-05-20T03:42:10.295Z] info bootstrap finished - 467.029 s
[2019-05-20T03:42:10.295Z] 
[2019-05-20T03:43:47.136Z] success Building production JavaScript and CSS bundles — 76.443 s
[2019-05-20T03:47:23.755Z] success Building static HTML for pages — 210.809 s — 55490/55490 276.87 pages/second
[2019-05-20T03:47:23.755Z] Generated public/sw.js, which will precache 12 files, totaling 402207 bytes.
[2019-05-20T03:47:23.755Z] info Done building in 782.100299522 sec
[2019-05-20T03:47:23.755Z] Done in 841.10s.
[2019-05-20T03:47:23.755Z] Done in 841.64s.

New:

[2019-05-24T04:15:58.931Z] success open and validate gatsby-configs — 0.010 s
[2019-05-24T04:16:03.113Z] success load plugins — 4.288 s
[2019-05-24T04:16:03.113Z] success onPreInit — 0.005 s
[2019-05-24T04:16:03.113Z] success delete html and css files from previous builds — 0.006 s
[2019-05-24T04:16:03.113Z] success initialize cache — 0.008 s
[2019-05-24T04:16:03.113Z] success start nodes db — 0.004 s
[2019-05-24T04:16:03.113Z] success copy gatsby files — 0.033 s
[2019-05-24T04:16:03.113Z] success onPreBootstrap — 0.014 s
[2019-05-24T04:16:03.113Z] Starting to fetch data from Prismic
[2019-05-24T04:16:03.371Z] Fetch Prismic data: 391.532ms
[2019-05-24T04:16:36.213Z] success source and transform nodes — 29.421 s
[2019-05-24T04:16:36.213Z] warning Deprecation warning - "noDefaultResolvers" is deprecated. In Gatsby 3, defined fields won't get resolvers, unless explicitly added with a directive/extension.
[2019-05-24T04:16:36.213Z] warning Deprecation warning - "noDefaultResolvers" is deprecated. In Gatsby 3, defined fields won't get resolvers, unless explicitly added with a directive/extension.
[2019-05-24T04:16:36.213Z] warning Deprecation warning - "noDefaultResolvers" is deprecated. In Gatsby 3, defined fields won't get resolvers, unless explicitly added with a directive/extension.
[2019-05-24T04:16:36.213Z] warning Deprecation warning - "noDefaultResolvers" is deprecated. In Gatsby 3, defined fields won't get resolvers, unless explicitly added with a directive/extension.
[2019-05-24T04:16:36.213Z] warning Deprecation warning - "noDefaultResolvers" is deprecated. In Gatsby 3, defined fields won't get resolvers, unless explicitly added with a directive/extension.
[2019-05-24T04:16:36.213Z] warning Deprecation warning - "noDefaultResolvers" is deprecated. In Gatsby 3, defined fields won't get resolvers, unless explicitly added with a directive/extension.
[2019-05-24T04:16:36.213Z] success building schema — 0.507 s
[2019-05-24T04:19:27.717Z] success createPages — 163.235 s
[2019-05-24T04:19:27.717Z] success createPagesStatefully — 0.071 s
[2019-05-24T04:19:27.717Z] success onPreExtractQueries — 0.004 s
[2019-05-24T04:19:27.717Z] success update schema — 0.000 s
[2019-05-24T04:19:27.717Z] success extract queries from components — 0.376 s
[2019-05-24T04:19:27.717Z] success write out requires — 0.291 s
[2019-05-24T04:19:27.717Z] success write out redirect data — 0.001 s
[2019-05-24T04:19:27.717Z] success Build manifest and related icons — 0.117 s
[2019-05-24T04:19:27.717Z] success onPostBootstrap — 0.119 s
[2019-05-24T04:19:27.717Z] info bootstrap finished - 200.574 s
[2019-05-24T04:19:27.717Z] success run static queries — 0.011 s — 2/2 194.43 queries/second
[2019-05-24T04:19:35.850Z] success Building production JavaScript and CSS bundles — 17.572 s
[2019-05-24T04:19:36.108Z] success Rewriting compilation hashes — 0.066 s
[2019-05-24T04:21:42.166Z] success run page queries — 122.355 s — 55588/55588 454.93 queries/second
[2019-05-24T04:24:40.233Z] success Building static HTML for pages — 157.017 s — 55588/55588 381.18 pages/second
[2019-05-24T04:24:40.233Z] Generated public/sw.js, which will precache 12 files, totaling 404337 bytes.
[2019-05-24T04:24:40.233Z] info Done building in 515.287176085 sec
[2019-05-24T04:24:40.233Z] Done in 578.07s.
[2019-05-24T04:24:40.233Z] Done in 578.55s.

KyleAMathews · 2019-05-28T00:25:44Z

Building JavaScript is faster as expected — interesting that running queries & SSR is so much faster 🤔

…

-- Kyle Mathews Blog: http://bricolage.io Twitter: http://twitter.com/kylemathews

On Mon, May 27, 2019 at 4:56 PM, Matthew Miller < ***@***.*** > wrote: @ KyleAMathews ( https://github.com/KyleAMathews ) Seems to just be generally all over the build: Old: [2019-05-20T03:34:23.683Z] success open and validate gatsby-configs — 0.015 s [2019-05-20T03:34:23.944Z] success load plugins — 4.015 s [2019-05-20T03:34:23.944Z] success onPreInit — 0.006 s [2019-05-20T03:34:23.944Z] success delete html and css files from previous builds — 0.007 s [2019-05-20T03:34:23.944Z] success initialize cache — 0.009 s [2019-05-20T03:34:23.944Z] success start nodes db — 0.005 s [2019-05-20T03:34:24.203Z] success copy gatsby files — 0.042 s [2019-05-20T03:34:24.203Z] success onPreBootstrap — 0.014 s [2019-05-20T03:34:24.203Z] Starting to fetch data from Prismic [2019-05-20T03:34:24.462Z] Fetch Prismic data: 387.842ms [2019-05-20T03:35:03.976Z] success source and transform nodes — 35.679 s [2019-05-20T03:35:03.976Z] success building schema — 1.061 s [2019-05-20T03:38:25.535Z] success createPages — 193.127 s [2019-05-20T03:38:25.536Z] success createPagesStatefully — 0.075 s [2019-05-20T03:38:25.536Z] success onPreExtractQueries — 0.027 s [2019-05-20T03:38:25.536Z] success update schema — 0.000 s [2019-05-20T03:38:25.536Z] success extract queries from components — 0.389 s [2019-05-20T03:38:25.536Z] success run static queries — 0.099 s — 3/3 30.93 queries/second [2019-05-20T03:39:31.986Z] success run page queries — 75.684 s — 55490/55490 736.00 queries/second [2019-05-20T03:39:33.397Z] success write out page data — 2.811 s [2019-05-20T03:39:33.397Z] success write out redirect data — 0.001 s [2019-05-20T03:42:10.295Z] [2019-05-20T03:42:10.295Z] success Build manifest and related icons — 0.257 s [2019-05-20T03:42:10.295Z] success onPostBootstrap — 0.260 s [2019-05-20T03:42:10.295Z] [2019-05-20T03:42:10.295Z] info bootstrap finished - 467.029 s [2019-05-20T03:42:10.295Z] [2019-05-20T03:43:47.136Z] success Building production JavaScript and CSS bundles — 76.443 s [2019-05-20T03:47:23.755Z] success Building static HTML for pages — 210.809 s — 55490/55490 276.87 pages/second [2019-05-20T03:47:23.755Z] Generated public/sw.js, which will precache 12 files, totaling 402207 bytes. [2019-05-20T03:47:23.755Z] info Done building in 782.100299522 sec [2019-05-20T03:47:23.755Z] Done in 841.10s. [2019-05-20T03:47:23.755Z] Done in 841.64s. New: [2019-05-24T04:15:58.931Z] success open and validate gatsby-configs — 0.010 s [2019-05-24T04:16:03.113Z] success load plugins — 4.288 s [2019-05-24T04:16:03.113Z] success onPreInit — 0.005 s [2019-05-24T04:16:03.113Z] success delete html and css files from previous builds — 0.006 s [2019-05-24T04:16:03.113Z] success initialize cache — 0.008 s [2019-05-24T04:16:03.113Z] success start nodes db — 0.004 s [2019-05-24T04:16:03.113Z] success copy gatsby files — 0.033 s [2019-05-24T04:16:03.113Z] success onPreBootstrap — 0.014 s [2019-05-24T04:16:03.113Z] Starting to fetch data from Prismic [2019-05-24T04:16:03.371Z] Fetch Prismic data: 391.532ms [2019-05-24T04:16:36.213Z] success source and transform nodes — 29.421 s [2019-05-24T04:16:36.213Z] warning Deprecation warning - "noDefaultResolvers" is deprecated. In Gatsby 3, defined fields won't get resolvers, unless explicitly added with a directive/extension. [2019-05-24T04:16:36.213Z] warning Deprecation warning - "noDefaultResolvers" is deprecated. In Gatsby 3, defined fields won't get resolvers, unless explicitly added with a directive/extension. [2019-05-24T04:16:36.213Z] warning Deprecation warning - "noDefaultResolvers" is deprecated. In Gatsby 3, defined fields won't get resolvers, unless explicitly added with a directive/extension. [2019-05-24T04:16:36.213Z] warning Deprecation warning - "noDefaultResolvers" is deprecated. In Gatsby 3, defined fields won't get resolvers, unless explicitly added with a directive/extension. [2019-05-24T04:16:36.213Z] warning Deprecation warning - "noDefaultResolvers" is deprecated. In Gatsby 3, defined fields won't get resolvers, unless explicitly added with a directive/extension. [2019-05-24T04:16:36.213Z] warning Deprecation warning - "noDefaultResolvers" is deprecated. In Gatsby 3, defined fields won't get resolvers, unless explicitly added with a directive/extension. [2019-05-24T04:16:36.213Z] success building schema — 0.507 s [2019-05-24T04:19:27.717Z] success createPages — 163.235 s [2019-05-24T04:19:27.717Z] success createPagesStatefully — 0.071 s [2019-05-24T04:19:27.717Z] success onPreExtractQueries — 0.004 s [2019-05-24T04:19:27.717Z] success update schema — 0.000 s [2019-05-24T04:19:27.717Z] success extract queries from components — 0.376 s [2019-05-24T04:19:27.717Z] success write out requires — 0.291 s [2019-05-24T04:19:27.717Z] success write out redirect data — 0.001 s [2019-05-24T04:19:27.717Z] success Build manifest and related icons — 0.117 s [2019-05-24T04:19:27.717Z] success onPostBootstrap — 0.119 s [2019-05-24T04:19:27.717Z] info bootstrap finished - 200.574 s [2019-05-24T04:19:27.717Z] success run static queries — 0.011 s — 2/2 194.43 queries/second [2019-05-24T04:19:35.850Z] success Building production JavaScript and CSS bundles — 17.572 s [2019-05-24T04:19:36.108Z] success Rewriting compilation hashes — 0.066 s [2019-05-24T04:21:42.166Z] success run page queries — 122.355 s — 55588/55588 454.93 queries/second [2019-05-24T04:24:40.233Z] success Building static HTML for pages — 157.017 s — 55588/55588 381.18 pages/second [2019-05-24T04:24:40.233Z] Generated public/sw.js, which will precache 12 files, totaling 404337 bytes. [2019-05-24T04:24:40.233Z] info Done building in 515.287176085 sec [2019-05-24T04:24:40.233Z] Done in 578.07s. [2019-05-24T04:24:40.233Z] Done in 578.55s. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub ( #13004?email_source=notifications&email_token=AAARLB6MPD3PPQS4XQWCASTPXRYKTA5CNFSM4HC2VINKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWKURIQ#issuecomment-496322722 ) , or mute the thread ( https://github.com/notifications/unsubscribe-auth/AAARLB4XFBQY5GMFWDIPSH3PXRYKTANCNFSM4HC2VINA ).

Moocar · 2019-05-29T03:39:39Z

OK, I'm satisfied again that this is ready to merge to master. I've published a new gatsby@per-page-manifest.

I've created the PR to merge per-page-manifest into master. #14359

I'm also running https://github.com/pieh/develop-runner to make sure that a bunch of sites at least build.

**Note: merges to `per-page-manifest`, not master. See gatsbyjs#13004 for more info** ## Description This PR saves the `page-data.json` during query running. In order to minimize the size of PRs, my strategy is to save the page-data.json along with the normal query results, and then gradually shift functionality over to use `page-data.json` instead of `data.json`. Once all those PRs are merged, we'll be able to go back and delete the static query results, jsonNames, dataPaths, data.json etc. It does mean that we'll be storing double the amount of query results on disk. Hopefully that's ok in the interim. Compilation-hash will be added in future PRs. ## Related Issues - Sub-PR of gatsbyjs#13004

Moocar · 2019-06-04T22:43:00Z

Hey folks, I'm closing this PR, as the actual code to be merged to master is on #14359 (branch per-page-manifest). Please comment there from now on.

**Note: merges to `per-page-manifest`, not master. See gatsbyjs#13004 for more info** ## Description This PR saves the `page-data.json` during query running. In order to minimize the size of PRs, my strategy is to save the page-data.json along with the normal query results, and then gradually shift functionality over to use `page-data.json` instead of `data.json`. Once all those PRs are merged, we'll be able to go back and delete the static query results, jsonNames, dataPaths, data.json etc. It does mean that we'll be storing double the amount of query results on disk. Hopefully that's ok in the interim. Compilation-hash will be added in future PRs. ## Related Issues - Sub-PR of gatsbyjs#13004

* feat(gatsby): Page data without compilation hash (#13139) **Note: merges to `per-page-manifest`, not master. See #13004 for more info** ## Description This PR saves the `page-data.json` during query running. In order to minimize the size of PRs, my strategy is to save the page-data.json along with the normal query results, and then gradually shift functionality over to use `page-data.json` instead of `data.json`. Once all those PRs are merged, we'll be able to go back and delete the static query results, jsonNames, dataPaths, data.json etc. It does mean that we'll be storing double the amount of query results on disk. Hopefully that's ok in the interim. Compilation-hash will be added in future PRs. ## Related Issues - Sub-PR of #13004 * Websocket manager use page data (#13389) * add utils/page-data.read * read websocket page data from utils/page-data * Loader use page data (#13409) * page-data loader working for production-app * get new loader.getPage stuff working with gatsby develop * fix static-entry.js test * remove loadPageDataSync (will be in next PR) * use array of matchPaths * Deprecate various loader methods * remove console.log * document why fetchPageData needs to check that response is JSON * in offline, use prefetchedResources.push(...resourceUrls) * root.js remove else block * loader.js make* -> create* * loader drop else block * pass correct path/resourceUrls to onPostPrefetch * s/err => null/() => null/ * Extract loadComponent from loadPage * remove pageData from window object * update jest snapshots for static-entry (to use window.pagePath) * add loadPageOr404 * preload 404 page in background * normalize page paths * comment out resource-loading-resilience.js (will fix later) * Remove data json from guess (#13727) * remove data.json from plugin-guess * add test for plugin-guess * Gatsby serve use page data (#13728) * use match-paths.json for gatbsy serve * remove pages.json * move query/pages-writer to bootstrap/requires-writer (#13729) And delete data.json * fix(gatsby): refresh browser if webpack rebuild occurs (#13871) * move query running into build and develop * save build compilation hash * write compilationHash to page data * reload browser if rebuild occurs in background * add test to ensure that browser is reloaded if rebuild occurs * update page-datas when compilation hash changes * use worker pool to udpate page data compilation hash * update tests snapshot * reset plugin offline whitelist if compilation hash changes * prettier: remove global Cypress * separate page for testing compilation-hash * fix case typo * mock out static entry test webpackCompilationHash field * consolidate jest-worker calls into a central worker pool * page-data.json cleanup PR. Remove jsonName and dataPath (#14167) * remove json-name and don't save /static/d/ page query results * in loader.cleanAndFindPath, use __BASE_PATH__. Not __PATH_PREFIX__ * loader getPage -> loadPageSync (#14264) * Page data loading resilience (#14286) * fetchPageHtml if page resources aren't found * add page-data to production-runtime/resource-loading-resilience test Also use cypress tasks for blocking resources instead of npm run chunks * fetchPageHtml -> doesPageHtmlExist * remove loadPageOr404Sync * revert plugin-offline to master (#14385) * Misc per-page-manifest fixes (#14413) * use path.join for static-entry page-data path * add pathContext back to static-entry props (fix regression) * Remove jsonDataPaths from pre-existing redux state (#14501) * Add context to sentry xhr errors (#14577) * Add extra context to Sentry xhr errors * Don't define Sentry immediately as it'll always be undefined then * mv cpu-core-count.js test to utils/worker/__tests__ (fix bad merge) * remove sentry reporting

KyleAMathews · 2019-06-11T06:39:30Z

For those following along here, this PR got merged via #14359 and released as 2.9.0 🎉 Thanks @Moocar for the great analysis and code and @wardpeet for all your reviews!

leonfs · 2019-06-11T08:50:54Z

Thank-you everyone!! amazing job!!

* feat(gatsby): Page data without compilation hash (gatsbyjs#13139) **Note: merges to `per-page-manifest`, not master. See gatsbyjs#13004 for more info** ## Description This PR saves the `page-data.json` during query running. In order to minimize the size of PRs, my strategy is to save the page-data.json along with the normal query results, and then gradually shift functionality over to use `page-data.json` instead of `data.json`. Once all those PRs are merged, we'll be able to go back and delete the static query results, jsonNames, dataPaths, data.json etc. It does mean that we'll be storing double the amount of query results on disk. Hopefully that's ok in the interim. Compilation-hash will be added in future PRs. ## Related Issues - Sub-PR of gatsbyjs#13004 * Websocket manager use page data (gatsbyjs#13389) * add utils/page-data.read * read websocket page data from utils/page-data * Loader use page data (gatsbyjs#13409) * page-data loader working for production-app * get new loader.getPage stuff working with gatsby develop * fix static-entry.js test * remove loadPageDataSync (will be in next PR) * use array of matchPaths * Deprecate various loader methods * remove console.log * document why fetchPageData needs to check that response is JSON * in offline, use prefetchedResources.push(...resourceUrls) * root.js remove else block * loader.js make* -> create* * loader drop else block * pass correct path/resourceUrls to onPostPrefetch * s/err => null/() => null/ * Extract loadComponent from loadPage * remove pageData from window object * update jest snapshots for static-entry (to use window.pagePath) * add loadPageOr404 * preload 404 page in background * normalize page paths * comment out resource-loading-resilience.js (will fix later) * Remove data json from guess (gatsbyjs#13727) * remove data.json from plugin-guess * add test for plugin-guess * Gatsby serve use page data (gatsbyjs#13728) * use match-paths.json for gatbsy serve * remove pages.json * move query/pages-writer to bootstrap/requires-writer (gatsbyjs#13729) And delete data.json * fix(gatsby): refresh browser if webpack rebuild occurs (gatsbyjs#13871) * move query running into build and develop * save build compilation hash * write compilationHash to page data * reload browser if rebuild occurs in background * add test to ensure that browser is reloaded if rebuild occurs * update page-datas when compilation hash changes * use worker pool to udpate page data compilation hash * update tests snapshot * reset plugin offline whitelist if compilation hash changes * prettier: remove global Cypress * separate page for testing compilation-hash * fix case typo * mock out static entry test webpackCompilationHash field * consolidate jest-worker calls into a central worker pool * page-data.json cleanup PR. Remove jsonName and dataPath (gatsbyjs#14167) * remove json-name and don't save /static/d/ page query results * in loader.cleanAndFindPath, use __BASE_PATH__. Not __PATH_PREFIX__ * loader getPage -> loadPageSync (gatsbyjs#14264) * Page data loading resilience (gatsbyjs#14286) * fetchPageHtml if page resources aren't found * add page-data to production-runtime/resource-loading-resilience test Also use cypress tasks for blocking resources instead of npm run chunks * fetchPageHtml -> doesPageHtmlExist * remove loadPageOr404Sync * revert plugin-offline to master (gatsbyjs#14385) * Misc per-page-manifest fixes (gatsbyjs#14413) * use path.join for static-entry page-data path * add pathContext back to static-entry props (fix regression) * Remove jsonDataPaths from pre-existing redux state (gatsbyjs#14501) * Add context to sentry xhr errors (gatsbyjs#14577) * Add extra context to Sentry xhr errors * Don't define Sentry immediately as it'll always be undefined then * mv cpu-core-count.js test to utils/worker/__tests__ (fix bad merge) * remove sentry reporting

Moocar added 30 commits April 2, 2019 07:50

documenting

071320f

Create queue configurations for build/develop

722c45a

moved page-query-runner to index

8868072

move src/internal-plugins/query-runner to src/query

d7dc21e

Also moved pages-writer.js and redirects-writer.js to src/bootstrap

add match paths json

6f036a5

write page data on query completion

330b2ba

run page queries after bootstrap

034cc78

use page data instead of data.json

9caff6a

add webpack compilation hash

d64e6bf

use compilation hash to refresh production-app

2b1b3c6

remove develop-html (use build-html)

64bd2b5

query-handler -> job handler. And ref ws manager from develop.js

f3292ff

invoke dev-404 correctly. Fix loader to handle develop mode

738fbb6

post merge fixes

854e7b2

only refresh page on compilation hash in production

54f79eb

get fetch 404 working

05fe38a

serve pages from gatsby develop

76e0b29

404 page is fully working. Introduced loadPage, getPage

f35ae9c

bump gatsbygram gatsby version

21aa5b8

misc fixes

7a12d71

fix 404 loader race condition

ae210a9

add dev loader for 404 pages

13204aa

deprecate loader functions (getResources...)

ae5ddab

app load pages in parallel

406f26d

cleaning up for PR review

21c325e

remove failed history stuff

8a0ed10

cleanup for PR

dcd65ea

standardize staticQueryResult.result

6dd6e1e

dataAndContext => pageData

6d7504e

Moocar mentioned this pull request May 27, 2019

Per page manifest #14359

Merged

wardpeet mentioned this pull request May 28, 2019

question - incremental builds support = part II #5002

Closed

Moocar mentioned this pull request May 29, 2019

revert gatsby-plugin-offline and update getResourceURLsForPathname #14385

Merged

Moocar mentioned this pull request May 30, 2019

Misc per-page-manifest fixes #14413

Merged

Moocar mentioned this pull request Jun 2, 2019

Remove jsonDataPaths from pre-existing redux state #14501

Merged

m-allanson removed this from In progress in OSS Roadmap Jun 3, 2019

KyleAMathews mentioned this pull request Jun 4, 2019

Missing resources for / #11524

Closed

Moocar closed this Jun 4, 2019

This was referenced Jun 11, 2019

fix(gatsby-plugin-offline): use networkFirst caching for page-data.json files #14720

Merged

fix(gatsby): Move importing of match-paths.json outside of loader.js #14732

Merged

m-allanson mentioned this pull request Jun 13, 2019

Don't build with webpack on content changes (no more data.json) #11982

Closed

afenton90 mentioned this pull request Jul 30, 2019

Can you opt-out of webpackCompilationHash (to improve Netlify build times) #15872

Closed

KyleAMathews mentioned this pull request Oct 29, 2020

Worse performance results with Lighthouse v6 (?) #24332

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Replace global data.json with page-data.json per page #13004

[WIP] Replace global data.json with page-data.json per page #13004

Moocar commented Apr 1, 2019 •

edited by bebrian458

Loading

me4502 commented May 24, 2019

KyleAMathews commented May 24, 2019 via email

me4502 commented May 24, 2019

KyleAMathews commented May 27, 2019

Moocar commented May 27, 2019 •

edited

Loading

me4502 commented May 27, 2019

KyleAMathews commented May 28, 2019 via email

Moocar commented May 29, 2019

Moocar commented Jun 4, 2019

KyleAMathews commented Jun 11, 2019

leonfs commented Jun 11, 2019

[WIP] Replace global data.json with page-data.json per page #13004

[WIP] Replace global data.json with page-data.json per page #13004

Conversation

Moocar commented Apr 1, 2019 • edited by bebrian458 Loading

Description

Benefits/Trade offs

Sup-PRs

Related Issues

me4502 commented May 24, 2019

KyleAMathews commented May 24, 2019 via email

me4502 commented May 24, 2019

KyleAMathews commented May 27, 2019

Moocar commented May 27, 2019 • edited Loading

me4502 commented May 27, 2019

KyleAMathews commented May 28, 2019 via email

Moocar commented May 29, 2019

Moocar commented Jun 4, 2019

KyleAMathews commented Jun 11, 2019

leonfs commented Jun 11, 2019

Moocar commented Apr 1, 2019 •

edited by bebrian458

Loading

Moocar commented May 27, 2019 •

edited

Loading