[1.0] Refactor and document APIs

[KyleAMathews]

One of the last big tasks before the first beta. Document our APIs and put them on a more solid conceptual foundation. I'll write up more on my plans Monday but wanted to get this PR started with my work so far. I got waylaid a bit building a transformer for Documentation.js to extract out metadata from jsdocs which... wasn't really necessary and took a while to get working but anyways, it'll be useful plugin all the same moving forward. I'll be writing the docs for our APIs using JSDocs and pull them out with the transformer for the website. See my tweet for a demo https://twitter.com/kylemathews/status/868478263378624512

I also finally made the file link inference step actually check if the field is pointing to an actual file. That was a recurring source of bugs (including one while working on this PR).

Lots more to come!

[gatsbybot]

Deploy preview ready!

Built with commit 385bc78

https://deploy-preview-1053--using-drupal.netlify.com

[KyleAMathews]

Deploy preview ready!

Built with commit 385bc78

https://deploy-preview-1053--gatsbyjs.netlify.com

[gatsbybot]

Deploy preview ready!

Built with commit 385bc78

https://deploy-preview-1053--gatsbygram.netlify.com

[0x80]

@KyleAMathews Is this related to the discussion in #858 also?

I'd still like to help out with that, although I am a little unclear on the whole api so documentation would probably help. Also there are a few things in the refactoring that I had in mind that could get in each others way. But if we make a list of things to do, possibly in order, then I could pick some up if you want me to.

Todo's I remember are

• create standard package entry point file for plugins
• optionally compile to dist, else reference src dir
• replace exports with explicit gatsby api calls

[KyleAMathews]

I thought about TODO three for a while and decided against it. exports.whatever is explicit and is just as statically analyzable as any other option. Also it's a lot simpler :-) as there's no boilerplate and it's already what exists.

I think optionally being able to export in a plugin's index.js the location of its gatsby-* files makes a lot of sense too but won't do that in this PR.

[0x80]

@KyleAMathews Alright! Let me know if there's anything in the API you would like me to contribute for 1.0 and otherwise I have enough other things to work on 🙂

[KyleAMathews]

thanks! I'll be posting a draft of the API design spec here Monday so would love feedback on that then.

[KyleAMathews]

Here's an draft for an API design spec. I'm working as part of this PR on converting all APIs and action creators to follow this design as well as adding this spec and initial API documentation to the website.

APIs have been growing fairly organically as needed while I've been working on v1 but before the first beta, I think it's important to systematize the design of APIs plus document them so we're on firmer footing moving forward.

The three main inspirations for this API and spec are React.js' API specifically @leebyron's email on the React API (https://gist.github.com/vjeux/f2b015d230cc1ab18ed1df30550495ed), this talk "How to Design a Good API and Why it Matters" by Joshua Bloch who designed many parts of Java. https://www.youtube.com/watch?v=heh4OeB9A-c&app=desktop, and Hapi.js' plugin design.

Gatsby's APIs are tailored conceptually to some extent after React.js which makes sense as users of Gatsby will necessarily also be users of React.js. I think there's a lot of wins to making Gatsby feel as familiar as possible as that'll ease adoption.

The two top priorities of the API are a) enable a broad and robust plugin ecosystem and b) on top of that a broad and robust theme ecosystem (themes are on the back burner btw until after v1 comes out).

Plugins

Plugins can extend Gatsby in many ways:

• Sourcing data (e.g from the filesystem or an API or a database)
• Transforming data from one type to another (e.g. a markdown file to HTML)
• Creating pages (e.g. a directory of markdown files all gets turned into pages with URLs derived from their file names).
• Modifying webpack config (e.g. for styling options, adding support for other compile-to-js languages)
• Adding things to the rendered HTML (e.g. meta tags, analytics JS snippits like Google Analytics)
• Writing out things to build directory based on site data (e.g. service worker, sitemap, RSS feed)

A single plugin can use multiple APIs to accomplish its purpose. E.g. the Gatsby glamor plugin 1) modifies the webpack config to add its plugin 2) adds a babel plugin to replace React's default createElement 3) modifies server rendering to extract out the critical CSS for each rendered page and inline the CSS in the <head> of that HTML page.

Plugins can also depend on other plugins. gatsby-plugin-sharp exposes a number of high-level APIs for transforming images that several other Gatsby image plugins depend on. gatsby-transformer-remark does basic markdown->html transformation but exposes an API to allow other plugins to intervene in the conversion process e.g. gatsby-remark-prismjs which add highlighting to code blocks.

Transformer plugins are decoupled from source plugins. Transformer plugins simply look at the media type of new nodes created by source plugins to decide if they can transform it or not. Which means that a markdown transformer plugin can easily transform markdown from any source without any other configuration e.g. from file, a code comment, or external service like Trello which supports markdown in some of its data fields.

See the full list of (official only for now — adding support for community plugins later) plugins at https://www.gatsbyjs.org/docs/plugins/

API

Concepts

• Page — a site page with a pathname, a template component, and optional graphql query and layout component
• Layout Component — surrounds a page and (eventually) can optionally have a parent layout component as well as a graphql query
• Template Component — responsible for rendering N pages. Can optionally have a graphql query
• Component extensions — extensions that are resolvable as components. .js and .jsx are supported by core. But plugins can add support for other compile-to-js languages.
• Dependency — Gatsby tracks automatically dependencies between different objects e.g. a page can depend on certain nodes. This allows for hot reloading, caching, incremental rebuilds, etc.
• Node — a data object
• Node Field — a field added by a plugin to a node that it doesn't control
• Node Link — a connection between nodes that gets converted to GraphQL relationships (is there a name for this?). Can be created in a variety of ways as well as automatically inferred. Parent/child links from nodes and their transformed derivative nodes are first class links.

Operators

• Create — make a new thing
• Get — get an existing thing
• Delete — remove an existing thing
• Touch — say an existing thing still exists
• Replace — replace an existing thing
• Set — merge into an existing thing

Extension APIs

Gatsby has multiple processes. The most prominent is the "bootstrap" process. It has several subprocesses. One tricky part to their design is that they run both once during the initial bootstrap but also stay alive during development to continue to respond to changes. This is what drives hot reloading that all Gatsby data is "alive" and reacts to changes in the environment.

The bootstrap process is as follows:

load site config -> load plugins -> source nodes -> transform nodes -> create graphql schema -> create pages -> compile component queries -> run queries -> fin

Once the initial bootstrap is finished, for the development server we start webpack-dev-server and a simple express server for serving files and for a production build, we start building the css then javascript then HTML with webpack.

During these processes there are various extension points where plugins can intervene. All major processes have a onPre and onPost e.g. onPreBootstrap and onPostBootstrap or onPreBuild or onPostBuild. During bootstrap, plugins can respond at various stages to APIs like onCreatePages, onCreateBabelConfig, and onSourceNodes.

At each extension point, Gatsby identifies the plugins which implement the API and calls them in serial following their order in the site's gatsby-config.js.

In addition to extension APIs in node, plugins can also implement extension APIs in the server rendering process and the browser e.g. onClientEntry or onRouteUpdate

[0x80]

@KyleAMathews It's looking good 👍

Although, I have to say I don't agree with your earlier comment

I thought about TODO three for a while and decided against it. exports.whatever is explicit and is just as statically analyzable as any other option. Also it's a lot simpler :-) as there's no boilerplate and it's already what exists.

I believe exports are not explicit because you can't easily detect typos and wrong use of api functions. You mention the HAPI plugin API as an inspiration, but there (like most plugin APIs) the plugin is a function with a well defined interface which gives you the API surface (server, options, next) to use. Any invalid use of the sever object will likely trigger an error and things like Flow can potentially warn you in advance.

If you type exports.onRoutUpdate no one will ever warn you that nothing will happen. And if Gatsby later changes a function name in the API, it'll have to keep checking for exports of the previous one, if you want to warn people about deprecating APIs.

I don't know any good API examples where plugins are defined using exports. Do you? Personally I think it sense to define plugins as a function and have the API passed in as is done in HAPI. The server object there could be the gatsby api where you can register for events or call synchronous functions.

const plugin = function (api, options, next) {
  api.onCreatePages(() => [])

  api.onPostBuild(args => {})

  api.modifyWebpackConfig(({ config, stage }) => {})
  return next() 
}

The next call at the end could enable plugins to do some asynchronous initialization before moving on to the next plugin. But this is just an afterthought I had looking at the HAPI interface 😄

I don't think this creates more boilerplate. I get that you want to release soon and the export API is already in place, but I still think this would be much nicer. If not for 1.0, then maybe for 2.0? 🙂

Another idea I just had for removing the gatsby-* file convention for plugins (previously discussed in #858) is to use the package.json to define the location of context related code. Like:

"gatsbyPlugin": {
    "ssr": "dist/ssr.js",
    "browser": "dist/browser.js",
    "node": "dist/node.js"
  }

Like that you don't have to enforce any convention for file names or their location, and plugins could choose to expose transpiled or original sources. Gatsby still controls when to load what file so it doesn't create conflicts for different execution contexts.

What do you think?

[jquense]

I tend to think exports works fine, I don't know why you couldn't validate it though, since it's just an object returned from a function (require)... I actual lying have a some work done in my console reporter branch that aggregates plugin api files up front during the load plugin bootstrap step. That would would be a fine place to validate to do some validation as the plugins get put in the store

[KyleAMathews]

@0x80 I think we have to keep an index.js as node's (so webpack's) resolution algorithm needs a main file in the directory or otherwise it'll throw an error that it can't find the module. Because of that I think leaving overriding the location of the gatsby-* files there makes the most sense. Either it's an empty file or there's the overrides.

[jquense]

I don't think you need a main file if you leave it off the package.json. I'd be surprised if webpack wasn't that flexible

[gatsbybot]

Oh well that would change things then :-) probably package.json is preferred then as we wouldn't ship extra bytes to the browser.

…

On Tue, May 30, 2017, 8:59 AM Jason Quense ***@***.***> wrote: I don't think you need a main file if you leave it off the package.json. I'd be surprised if webpack wasn't that flexible — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1053 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AYdmDVsB_fBBI71A_rTOHL1O411dhhHzks5r_DzmgaJpZM4NooAJ> .

[jquense]

any reason to even allow overriding the locations? seems like a nice place to save on complexity with a fairly uncontroversial convention.

[KyleAMathews]

I think the main reason is to make it easier to ignore built files #858 (comment)

[KyleAMathews]

Deploy preview failed.

Built with commit 385bc78

https://app.netlify.com/sites/image-processing/deploys/592e2a286f4c5024628005c9

[sillyslux]

oh look @jquense is here!!
@KyleAMathews will i, in 1.0, still be able to build with option noProductionJavascript?
b/c react-bootstrap/react-bootstrap#2510
i've even created a fork for that

[KyleAMathews]

@sillyslux I'm not interested in supporting a no-js version of Gatsby. It doesn't make any sense really. If you for some reason really don't want to load JavaScript, just don't include the script tags in your html.js file.

[sillyslux]

So the option will be gone, but i can still run my own prod-js, like i already did before i've discovered noProductionJavascript?
good-enough for me...
But on the other hand, just for clarification, if i build a large wiki with it, gatsby's bundle.js grows with
even only more content and no changes to it's JS, is that right?
For just a simple SPA like behaviour i already use a gh-pages hack with react-router somewhere else. I don't see the point in tons of html files when users ultimately don't even get them served.

[KyleAMathews]

@sillyslux no v1 does code and data splitting by default so you only load what's needed for a given page. We are still loading a "routes" map for the entire site which will grow (slowly, each route needs very little data) but it's quite possible to lazy load routes as well which will get tackled post 1.0. #431

[sillyslux]

Now that's awesome news.
But in the end i will still have to choose between a bundle.js (500KB) and a custom.js (5KB)?
Sorry, that was a rhetorical question, to sum things up.

[KyleAMathews]

The default bundle has ~70kb. Beyond that it's whatever you add. But sure, if you're very concerned about the number of bytes you're shipping then remove the JS. But it's not supported because it's a significantly poorer experience given no client-side routing + it breaks any interactivity you might want to add.

[KyleAMathews]

The new action creators docs page is coming along https://deploy-preview-1053--gatsbyjs.netlify.com/docs/action-creators/

Gatsby + Documentation.js is really nice for writing docs!

One bummer is I wasn't able to figure out how to use flow instead of JSDocs. Probably is possible just I didn't spend enough time on it. At some point let's move things over.

[sillyslux]

Does lazy-loading mean it will load HTML files for new page-views or is it JS then? I'm kind of concerned i might loose the html...
I use gatsby to create a static site from some repo's master branch, into it's gh-pages branch with prettified html and i really don't want it to do any kind of SPA things for me. I chose gatsby over jekyll and hope i can keep gatsby's input- and output-files as minimal as they are now.

[KyleAMathews]

@sillyslux Gatsby is a universal JS framework meaning it renders its HTML using react.js components on the server and client. So in the client, it only loads the HTML for the initial page. For other pages it lazy loads in the component bundles + necessary data to render them.

[sillyslux]

In it's description it says: "Blazing fast React.js static site generator".
Is that about to change?

[KyleAMathews]

Deploy preview failed.

Built with commit 6acd904

https://app.netlify.com/sites/using-postcss-sass/deploys/592e18078ebdd9191ea217ab

[KyleAMathews]

Some of the docs are a bit of a rush job but overall feeling pretty good about them! Nice to have docs finally! Plus good foundation for improving them in the future.