Preserving backwards compatibility when adding new keywords

Question

Inspired by Why do keywords have to be reserved words?

Suppose that you're the BDFL of a programming language. Version 1 of the language becomes decently popular. A few years later, you decide to make Version 2 with a bunch of new features. This requires new keywords to be added to the language. For the sake of making the example concrete, let's suppose that the new keywords are async and await.

Problem is, there are programs written in Version 1 of the language that happen to use async or await as the name of a variable or function. And now that these are reserved words, the existing code no longer compiles.

What approaches do existing programming languages take to minimize the hassle of renaming identifiers when they conflict with newly-added keywords? How would you choose to handle this situation?

Answer 1

Require explicit opt-in to language version

Taking inspiration from C#, one can require programs declare which version of the language they are using. Then, the compiler/interpreter may choose to only handle select newer features depending on declared version.

An example is the required keyword, which in programs written for <=C#10 will warn when attempting to use it, and >=C#11 will refuse to accept as an identifier.

This allows older programs to compile and run fine on a newer compiler/interpreter, but newer programs may opt in.

If version is declarable at the file or scope level, then new keywords can be used in an older program without a total rewrite.

Answer 2

Let's suppose that the new keywords are async and await. What approaches do existing programming languages take to minimize the hassle of renaming identifiers when they conflict with newly-added keywords? How would you choose to handle this situation?

I did have to handle that situation, so I can tell you exactly what we did.

In C# 6 await means the operator only in methods that are declared with the async modifier. In C# 5 and before it was a syntax error to use async as a modifier, so there was no back compat issue; it is not a breaking change to turn a program that never compiled into one which does. Moreover, async as a modifier is legal only in a declaration; every other usage can be treated as an identifier, so again, no back compat issues.

That is, async had no semantic purpose in C# 6. It is only to signal to the compiler that await means the operator in what follows.

More generally, we used several different tricks throughout the evolution of C# to add new keywords. First off, it's important to realize in C# that there are "reserved" and "contextual" keywords. C# has only added contextual keywords since C# 2. (And C# 1 started with contextual keywords; get and set are contextual for example.) Also, C# allows any identifier to be preceded by @ to indicate that it is to be treated as an identifier, not a keyword, even if it is reserved.

Every time a new keyword was added the grammar and/or semantic analysis pass was carefully designed so that we could determine whether the keyword was being used in its new sense, or was an identifier. For example when yield was added to C# 2 the original design was just yield whatever; as a statement, but that is grammatically ambiguous with declaring local whatever of type yield, so the design was modified to require yield return whatever;, which was always a syntax error in C# 1.

You've probably noticed that this choice for C# 2 was different than the choice for C# 6; we could have in C# 6 chosen a two word phrase for the operator instead of requiring a marker modifier. It could have been async wait whatever; and that would have been a syntax error in previous versions of the language. We considered many options like that before arriving on the marker modifier solution; consistency with yield return was attractive but ultimately was considered too wordy. (Whether the C# 2 design team considered a marker modifier for iterator blocks, I do not know offhand.)

When designing LINQ for C# 3 there were a great many contextual keywords added and it was not clear what query syntax we'd arrive at. To facilitate finding problems early one of our developers with expertise in parser theory wrote a grammar analyzer which could produce examples of expressions that parsed ambiguously for a given query grammar proposal. That certainly helped us find lurking problems early in the design process.

In C# 4, dynamic was easy. There's no special grammar logic needed, and the semantic analyzer just checks to see if dynamic already has a meaning in the program and if it does, it uses that, otherwise it falls back to the special type. Same for var in C# 3.

I hope that helps; let me know if you want more details on any of these techniques.

Answer 3

Depending on the syntax specifics, a new keyword can be contextual at the expense of a new ambiguity the parser must handle.

For example, let's say you add a prefix unary expression operator await to your language, and there's a program with an await identifier:

var await = 4;

This need not be an error since it is not a legal use of await anyway. In fact, in most positions where an identifier is declared, a keyword is not going to be ambiguous. Later, in other expressions, when the parser encounters a sequence like

var x = await

it needs to do one (or possibly more) token of lookahead to see what's after it. For example, you might have

var x = await * 2;

The * token is enough to disambiguate.

Problems may arise if a certain token forms both unary and binary expressions, for example

var x = await + await;

This might be await (+await), or (await) + (await), depending on your rules. In this situation you might choose to simply require parens to disambiguate, choose the new or old behavior consistently, or defer to other strategies as described in other answers.

Answer 4

Provide an upgrade tool

Another approach is to not preserve backwards compatibility and instead provide tools to upgrade to the new version.

In the case of adding a couple of new keywords all the upgade tool would have to do is rename variables using the string that is now a reserved word.

Make it possible to identify the language dialect in use

This procedure is facilitated if you have something in the source code that can identify the version in use. Otherwise your tool may incorrectly treat the new keywords as variables.

Example

For example running foo1to2 on:

dialect(foolang,1)

var IamAKeyword : integer = 2;

It becomes:

dialect(foolang,2)

var IamNotAKeyword : integer = 2;

Upgrade tools can be a pain to write if the nature of the change is complex or the syntax is complex. You can make your life easier if you consider this issue upfront.

Consider context

As others have said you can somewhat avoid the issue if your new syntax is illegal in the previous language version/dialect.

Beware though that you are trading convenience of upgrade for clarity in the language.

An interesting example of this is C++. Relatively early on for pure methods the = 0 syntax was introduced rather than a pure keyword

class Foo
{
   virtual void someMethod() = 0;
};

assignment of methods was not a legal syntax. Much later when override and final were introduced they appear in the same place. So instead of:

class Foo
{
   override void someMethod();
};

Where override could seem to be the name of the return type say You write:

class Foo
{
   void someMethod() override;
};

There was a good link about this which I cannot find but https://stackoverflow.com/questions/32757571/why-do-the-c-language-designers-keep-re-using-keywords is very relevant.

Don't bother - just document

A final option is to not to anything other than document carefully the changes.

You leave the community deal with the issue. They may well come up with upgrade tool on your behalf.

This risks alienating your user base, particularly if you do it too often. But if your language is good and your community loyal you can get away with it. This is really trading user friendliness for development resources.

There are quite a few languages that are guilty of this (despite having ample developement budget) and remain popular.

Deprecate

When evolving a language you could consider formally deprecating interfaces in one version before removing or changing them in a subsequent version. This slows down changes to your langauge but makes it easier for users to manage the transition.

This is particularly useful in 'enterprise' environments.

You can also consider having a syntactic or semantic way to mark something as deprecated in the language itself. For example an attribute like [[deprecated]] in C++ can be used to indicate a function is to be retired.

This may or may not be harder to do for a keyword. In some languages keywords are a lot like functions or even exactly the same (e.g. TCL.

My personal preference is both:

1. a clear way to the dialect/version used.
2. an upgrade tool

Answer 5

Stropping was the approach taken in some early programming languages like Algol-60 and Algol-W. The Wikipedia Article says

In computer language design, stropping is a method of explicitly marking letter sequences as having a special property, such as being a keyword, or a certain type of variable or storage location, and thus inhabiting a different namespace from ordinary names ("identifiers"), in order to avoid clashes. Stropping is not used in most modern languages – instead, keywords are reserved words and cannot be used as identifiers. Stropping allows the same letter sequence to be used both as a keyword and as an identifier, and simplifies parsing in that case – for example allowing a variable named if without clashing with the keyword if.

Algol-60 implementations frequently used single quotes around a keyword, e.g. 'BEGIN'. On some machines it used underlining. And in the language specification this was typically represented as bolding.

Algol 68 used several alternate strapping conventions, that could be specified by a compiler directive, like .FOR, 'FOR, upper/lowercase... plus they had the option of not requiring strapping at all, and just requiring reserved words like most modern languages. In publication keywords were typically represented as underlined or bold.

FORTRAN has limited stropping for reserved words or operators like .EQ. and .AND.

Stropping went out of fashion as programming language portability became more and more important, since the least common denominator for composing programs was in plain ASCII text with little or no formatting. The usual mechanisms of displaying stropped keywords via fonts like bold could not be relied on to be portable.

Stropping avoids breaking existing programs if new keywords collide with identifiers in old code. Stropping avoids the contortions that languages like C++ need to provide for extensibility, avoiding such conflicts. Nearly all modern programming is done using IDEs or editors that can do syntax coloring. Syntax coloring is almost exactly what you need to do to make stropping user-friendly. You don't need to require the user to type the stropping syntax: the user can type in plaintext, and the editor/IDE can apply stropping automatically, and save the stropped format in a file. If new syntax is added with a new keyword foo that happened to be used as an identifier foo in old programs, no problem. There are several portable formats that can save such stropped program representations, if necessary with more metadata than simple .stropping. can represent.

However, stropping does not completely eliminate problems with extensibility. E.g. if you wish to design a language that allows new language constructs, syntax, and keywords to be added to your language. If there are multiple such extension packages, they can conflict. Ultimately you will need to have some sort of namespace indication for such syntactic extensions. So you might want to go beyond simple .stropping., who have representations that can store more metadata.

Answer 6

A quick and dirty approach C uses is by reserving all identifiers starting with _, and using _Keywords as built-in keywords and only having the keywords as convenience macros in header files.

Answer 7

Use namespaces everywhere

This could mean many things. You may choose what is appropriate in your language:

1. A user-defined identifier could override the meaning of a keyword.
2. You could use the original keyword by explicitly specifying the namespace, for example system::await.
3. You could alias a keyword to a different name in a program.
4. You could import the namespace only if you need the keyword in your program.

It might be easiest to implement only #4, in which case the namespace may not be called a namespace, but a feature option that you could turn on and off. But there would be problems if it exported identifiers that code using new features would use. You could support aliases in the import, but that wouldn't work well for members in a structure. But there is an easy solution, by using a special grammar for forcing the use of an identifier that is the same as a keyword. You could turn off the option in old source code files, and when you want to upgrade a file, change the identifiers with conflicting names to the new grammar. C# uses @x. Some SQL dialects use `x` or [x]. Shell scripts, say bash, uses 'x' or any other quotes or escapes, and also command x which also avoids functions. PHP uses ${'x'} for names that is not in the format of an identifier, but may not be the best example because variables and keywords are not in the same namespace in the first place.

Answer 8

Different lexical forms for keywords and identifiers

In my current design, identifier names must contain a capital letter, digit or underscore; anything that is purely lowercase letters is inherently a keyword. There is thus no risk of a variable name becoming a reserved word in a future version, because it can't be a valid variable name in the first place.

Answer 9

You can add syntax to tell the parser which language standard to use. Ada has that.

But, in most cases it is fine to warn the user upfront and use keywords that most people expect to be keywords anyway.

Answer 10

Python had exactly this issue (specifically for async/await), and ended up introducing those as 'soft keywords'. Essentially, there's no special case for them in the lexer, so they are treated as identifiers like any other, but there are also grammatic rules that expect these specific identifiers in specific positions. Since the new grammatic constructs would've been illegal previously anyway, no collisions take place. The only (slight) downside to this approach is performance of parsing, but the effect is miniscule

Answer 11

I think that the possibility of adding new keywords reinforces the idea that they should either be a property of the base language ab initio, or should be identified in some way. As somebody else has mentioned, ALGOL mandated or at least permitted stropping, while in Modula-2 Wirth mandated that keywords were capitalised while at least preferring that non-keywords (i.e. imported from libraries etc.) were not.

I would suggest designing the base language such that a spurious non-keyword identifier can always be recognised as an error, and that pragmata are used to indicate e.g. storage class rather than keywords.

From there, I would suggest first providing an intermediate version of the language, where any existing usage of the new keyword is flagged as an error which either has to be fixed or blessed by a pragma indicating that it's been reviewed.

Putting keywords in their own namespace, or requiring that a program unit be imported to make them available, is one approach. But my own feeling is that they should be minimised, in order- apart from anything else- to reduce lexing problems caused by the demands of internationalisation.

Answer 12

Make sure those keywords can only appear in places where program-specific identifiers can not. That way there can be an identifier that is spelled like a keyword without their being any ambiguity. This can be difficult in a C-style language where identifiers can appear in all kinds of places. But it is far more achievable in a COBOL-style language where every line begins with a keyword that says how the rest of the line is to be parsed. For example, when your syntax for calling a function looks like that:

FUNCTION async TAKING input TYPE string
   WRITE input
ENDFUNCTION

CALL FUNCTION async PASSING await

then you could in the next vesion allow this:

ASYNC FUNCTION async TAKING input TYPE string
   WRITE input
ENDFUNCTION

AWAIT FUNCTION async PASSING await

When every line must start with a keyword, then you can add as many more line-starter keywords as you want without there ever being a conflict.

Answer 13

Preprocessor magic

One approach that can be used if your language has a token stream to token stream preprocessor, is to utilize it in the keyword-or-identifier decision making process.

The basic idea is to change the lexer to always output keyword or identifiers as identifier tokens¹, leaving it up to the preprocessor to replace identifier tokens representing a keyword with a keyword token.² This makes it possible to have preprocessor directives that can affect the identifier or reserved word decision. For example, you could have a preprocessor directive that turns on or off a set of keywords, or even an individual keyword. Doing it as a matched set of directives (akin to #if and #endif) can be especially useful for scoping these changes, especially for code that may be #included and not know what state to return to.

The main advantage of this versus a language version compiler flag, is that you can write code that uses the new keyword for its new purpose in the same file, simply by wrapping the section that uses it with a preprocessor directive.

The only language I am aware of that does this is Verilog, which supports a `begin_keywords preprocessor-like directive, and a corresponding `end_keywords. The `begin_keywords directive takes an argument that indicates the standard revision that defines the keywords to support.

---

Footnotes:

¹ Traditionally a lexer will output special tokens for keywords, making them trivial for the parser to identify. Having to look inside a token to decide the parse path is more complicated, and often not supported by parser generators if your language is using one of those.

² Obviously, there are some variations possible, like always output all reserved words as their own tokens in lexer, and having preprocessor conditionally convert them back to identifiers, etc.

Answer 14

Reserve unused keywords for future use

Most of the time a new keyword is added to a language after its initial release, the keyword is one which exists in other languages already. The obvious examples are async, await and yield, but e.g. const and let were added to Javascript, case and match were added to Python, and so on.

This means that with some foresight, it's possible to reserve some keywords before the associated features are added to your language; then when it comes to add those features, there is no backwards compatibility problem because the keywords you need have already been reserved from day one.

Examples:

• In Rust, the keywords abstract, become, box, do, final, macro, override, priv, typeof, unsized, virtual and yield are currently reserved for future use.

• In Java, the keyword _ is reserved for "possible future use in parameter declarations"; and const and goto are reserved to allow for better error messages in case C++ programmers use these by mistake, but it would also be possible for Java to use them for new features in the future.

• Nim likewise has some unused keywords reserved for future use, but I didn't find a comprehensive list of which ones are currently unused.