I’m Ham and I’m a developer on the Teams team here at Stack Overflow. Over the past few months, I’ve been heads down working on the way we turn Markdown into HTML when writing and editing posts across the network. I’d love to share what I’ve come up with.
In a nutshell: We're planning to use CommonMark for all posts across the network moving forward. To do so, we switch to CommonMark-compliant Markdown renderers on the client and the server side. We have to make sure that all existing posts work with the new renderers so we will run a big migration across the network that will convert existing posts to use the new CommonMark format. Writing, editing and reading posts should look and feel mostly the same after the change.
As of June 20, 2020, all sites are on CommonMark now. For individual sites, see the migration schedule here.
We’re using Markdown throughout the Stack Exchange network. Markdown was one of the early technology bets when Jeff and Joel started out building Stack Overflow. If you write a question, an answer or a comment anywhere on the Stack Exchange network, you’re going to write it in Markdown.
Over the years, Markdown has become a common way of writing content in online communities. It has become a wild success and even got a formalized specification with CommonMark.
Stack Exchange’s way of handling user-created Markdown today is largely the same as it was when we started. We’re using our own, home-grown Markdown parsers and renderers on the client and server side. Both of these implementations have proven to be a solid foundation and received a lot of tweaks over the years.
However, they come with their own quirks. Being created before there was a CommonMark spec they show some non-spec-compliant behavior. They’re using regular expressions for transforming Markdown into HTML (I’ll leave it to your imagination how much sweat and tears this has cost us over the years) which is perfectly doable but makes maintaining and adding new features to our Markdown parsers extra hard.
The idea
We think it’s time to move forward. You’ve asked if we’re ever going to adopt CommonMark on the Stack Exchange network a few years back, balpha ran the numbers, and while he found out that it wasn’t impossible, it didn’t seem to be easy peasy, either. With some of the past and upcoming changes, we think that now’s a great time to tackle this challenge and to migrate all network posts over to CommonMark. This includes:
- Changing the Markdown renderer on the client side
- Changing the Markdown renderer on the server side
- Automatically editing and re-rendering all posts across the network that are not CommonMark-compliant
To give you a better feeling for the changes under the hood: When you write a post on the Stack Exchange network, you write it in Markdown. On the client side, you see a preview of your post as you’re writing it. This preview is created by our client-side Markdown renderer. It takes the Markdown you write, transforms it into HTML and shows you a preview of what your post will look like.
Once you save your post, we send your Markdown over to our servers where the same Markdown-to-HTML conversion takes place, again (you can’t trust user input, so we don’t blindly accept the HTML generated on the client side).
Our plan
We will migrate sites across the network to CommonMark site by site over the next couple of weeks. We plan to start with Meta Stack Exchange and Meta Stack Overflow on Wednesday, June 3rd 2020.
I’ve prepared a feature that will swap out our current, home-grown Markdown renderers with well-tested open-source implementations that adhere to the CommonMark specification. For the curious: this means we’re replacing PageDown with markdown-it on the client side and MarkdownSharp with markdig on the server side.
Once we enable that feature, new and edited posts will automatically be rendered with those new renderers. Most likely, you won’t even notice a difference when looking at posts.
With the new renderers in place, we’re going to move all existing posts across the network over to CommonMark. For the vast majority of the posts across the network (80% and up), this means nothing will change. Most of the posts on our network have been written in a way that is completely compliant to the CommonMark specification already, yay! If we convert this Markdown to HTML using a new renderer, the results will be exactly the same.
Then there are those posts that are written in a Markdown flavor that was cool for our current renderers but isn’t what CommonMark would expect. Balpha’s analysis is giving you more details on that. We’re talking about ##headlines without spaces after the hashes and other minor oversights. For these posts, we’ve built a tool that automatically fixes these well-known issues by changing a post’s Markdown source directly and re-rendering the HTML of the post in question. When we change a post’s Markdown automatically, this will end up looking like a regular edit but we’re making sure that this won’t bump posts to the top.
So now we’ve got about 80% of our posts that are already good to go. With the auto-fixing utility we estimate that we’re going to land at over 96% of all network posts being rendered completely identical after migrating to CommonMark and using the new renderers. This leaves us with a few percents of posts that end up looking differently when rendered with the new CommonMark renderers.
What you can expect
We avoid breaking existing posts by erring on the side of safety. If a post looks different using the new renderer (and if it’s just one whitespace off) we won’t automatically re-render the post and put it up for investigation first. This way we can be sure that all changes are safe.
I’ve played around with our data to get a feeling for the posts that will be rendered slightly different after using the new renderers. I found out that the differences fall into three buckets:
- False positives: the HTML markup changed slightly but doesn’t change semantics or presentation of the post
- Improvements: things where the CommonMark specification fixes some oversights in our current Markdown flavor
- Actual issues: things that we didn’t anticipate and need to fix
The "actual issues" category should be a tiny fraction but I won’t naively assume that they won’t happen. There will be some changes caused by the new Markdown renderer that we need to investigate because they will cause posts to look different than before in one way or another. We can’t foresee all edge cases that this change will introduce so we will surface all posts that look different when rendered with the new Markdown renderer, review them and if necessary fix them.
Ultimately, we hope to make this transition as smooth and frictionless as possible. We don’t want to break (and manually fix!) thousands of posts. We don’t want to mess with your writing experience. At the same time, I know that we won’t get this feature perfect from the get-go, so I need to ask for your patience and understanding.
As you write new posts after we’ve made the switch to the new CommonMark renderers, you will have the exact same writing experience as before. The preview will show you what your post will look like and once you save your post, it should appear just as you saw it in the preview. If you notice any differences between preview and saved post, please let us know!
Things might get funky when you're editing a post that renders differently with the new CommonMark renderer. Again, if we detected that a post would look differently when rendered with the new CommonMark renderer during the migration, we wouldn't save a new version of this post as part of the migration. This way, all posts continue to look the same when being viewed. However, once someone comes in and edits it, it will be rendered using the new CommonMark renderer and this might cause the post to look slightly different than what we had before. This will only be a small fraction of all of our posts, and of that small fraction a smaller fraction will actually be edited moving forward. However, it's important to keep in mind that editing old posts has a slight chance that you run into differences between our old and our new markdown renderers.
Frequently Asked Questions
When is this going to happen?
The new CommonMark renderers are being merged into master within the next few days. They’re hidden behind a feature flag, so they won’t do any harm until we flip the switch.
We will migrate sites across the network site-by-site over the course of the next couple of weeks. We will start with Meta Stack Exchange and Meta Stack Overflow on Wednesday, June 3rd, 2020 (assuming everything goes well and we don't discover a major blocker until then). Since we can't exactly predict what kind of dragons we'll encounter along the way, plans might change slightly. I'll post a plan for sites and their switchover dates as an answer to this question and will keep it updated as we go.
Every site is different and we need to learn as we go. Most sites can be migrated within a few hours. For our biggest network sites, changing all posts to CommonMark will probably take up to 4 days. Keep an eye on the schedule I'll post to see how we're doing.
Why are we migrating to CommonMark?
In the past, changes to our Markdown renderers have been rather risky and high-effort. We needed to carefully evaluate if a change breaks anything for the millions of existing posts we have in place. By sticking to a well-defined specification like CommonMark, we can make sure that implementations that stick to this specification will work for us. If the specification gets extended, adopting changes will be easy and safe.
Another reason is that this reduces some of the maintenance burdens of our development teams. Instead of maintaining two distinct Markdown renderers, we can now pick something off the shelf and use that instead. With markdig and markdown-it we’ve found two reputable libraries that are beating our own implementations when it comes to performance and functionality. Both are great pieces of software that we're more than happy to use in our product.
Are there some changes to the way I can write Markdown in the future?
Yes, there will be a few changes to the set of supported Markdown on Stack Exchange. For the vast majority of your writing, you won’t see a difference at all. We’re doing our best to continue to allow most of the syntax you can use right now. We’re adopting the CommonMark standard, so everything that’s valid CommonMark will work on Stack Exchange moving forward (here’s a short cheat sheet for the curious).
At the same time, we want to take this opportunity to remove some quirks we’ve built during times where there was no such thing as a CommonMark standard. Some features in Stack Exchange’s current Markdown flavor have been built during a time where there was no standardized way of doing things. Now that we’re adopting CommonMark, we want to replace some of those homegrown features with standardized notation, a notation that you know from other places all around the web as well.
The most noticeable changes will be around lists, nested lists, headlines and blockquotes.
Lists: When creating nested lists, you’ll need to indent your nested list items or paragraphs with the right amount of spaces. While one space was enough before, you’ll need to add a few more now, depending on your type of list.
To make a paragraph part of a list item it used to be enough to add one space in front of the paragraph
* this is a list item
that goes on here
with CommonMark, the paragraph has to line up with the text of the parent, so we need a few more spaces here:
* this is a list item
that goes on here
Headlines:
Moving forward, you have to add a space after the leading # characters.
#this was cool before
# this is what's cool now
Blockquotes:
Previously, empty lines between two lines marked as blockquotes would make one big blockquote. Moving forward, you'd get two distinct blockquotes this way, unless you start the empty line with a > character, too:
> old blockquotes
> with multiple lines
> new blockquotes
>
> with multiple lines
Indented code block highlighting: Our biggest breaking change will be around indented code blocks and the possibility to declare the language to be used for syntax highlighting.
In a nutshell: If you want to declare the language for syntax highlighting in your code block, use the code-fence notation and not indented code blocks. You can still use indented code blocks, but declaring the preferred language explicitly for them is no longer supported moving forward.
Until now, you could do this to declare the language for an indented code block:
<!-- language: python -->
def hello():
print("Hello, World");
Moving forward, this style is considered deprecated. Ever since we’ve introduced code fences, you can explicitly declare the language of a code block using the code fence notation:
``` python
def hello():
print("Hello, World");
```
This is the way the CommonMark standard is proposing and this is what other websites are doing, too. We know that you might have gotten used to using the old syntax featuring a <!-- language: lang --> comment. As we’re adopting new Markdown parsers, we want to avoid patching quirky behavior into those parsers when there’s an official, standards-compliant way of achieving the same goal that we can adopt with no extra effort. This style will continue to work for the time being, but is subject to removal in the future, at which point posts using it will no longer recognize it.
Note that setting the syntax highlighting language based on the tags you’ve associated with your post will continue to work. Here’s a full overview of the current behavior of our syntax highlighting if you need a refresh. We’re going to update that post as we move forward.
What happens to SE-specific syntax elements?
On the Stack Exchange network we support some syntax elements that are not part of the CommonMark standard. Things like spoilers, MathJax, circuit diagrams, stack snippets, etc. are used on several network sites. We're going to continue to support all of those custom syntax elements even if they're not part of the official CommonMark specification.
Will this finally enable table support?
Maybe! Support for tables has been discussed intensely in the past. There are many creative workarounds out there but never any official support for rendering tables. If other sites can pull this off, why can’t we?
One of the main reasons, the fact that our Markdown parsers and renderers didn’t support tables, is now no longer valid as we’ve switched to Markdig and Markdown-it. Both support parsing and rendering tables out of the box. Still, introducing table support is a change we don’t want to blindly stuff into this big migration.
Let’s get everything to work nicely with the official CommonMark specification first – and just to be clear, tables are not part of that spec. This change is massive; we need to see how this plays out and make sure this doesn’t introduce more than a few acceptable cosmetic issues across all our communities.
Once the dust has settled and we’re all comfortable with the new Markdown renderers under the hood, we can re-evaluate if the time is right to bring table support back to the (drumroll) table!
[...], such as tag ([tag:discussion]for discussion) or site reference ([scifi.se]for Science Fiction & Fantasy)? Are those just modelled as links whose definition is invisible at the time of writing?>you'd start each line of a nested blockquote with>>(or more characters if you want to nest even deeper).