LaTeX3 versus pure Lua

Question

Reading this site for the last nearly two years I've learned a lot about LaTeX3 (which I had previously understood to be a defunct pipe dream) and LuaTeX (of which I was previously totally unaware). As I have said before, TeX itself is a bad programming language, though a good typesetting engine, but even as a typesetting engine, it is hobbled by its limitations as a language. What I mean by both of these, in short, is that it does not provide an easy way to manipulate its own representations of data. Others have also had this problem, and sometimes it can be solved with LuaTeX, which opens up a lot of the internals to the programming environment.

Now, as I understand it, LaTeX3 also provides a lot of programming tools within TeX, with the aim of making package creation and maintenance more robust. Joseph Wright has explained this in at least one place (and therefore probably many more). These are "algorithmic" tools, while the direction of LuaTeX development appears to be "data structure" tools. Of course, Lua is also a general-purpose programming language and as such provides perfectly good algorithmic tools as well.

My question:

Why use LaTeX3 programming over Lua, when LuaTex is available?

I understand that LuaTeX is very much experimental now, but then, so is LaTeX3. I also understand that as a result, LuaTeX is not necessarily used by anyone "ordinary", but then, it is intended as the ultimate successor to pdfTeX, which is now the default engine in the major TeX distributions. For all the reasons that TeX is a terrible programming language, LaTeX3 will still suffer from irritating expansion issues, lack of arithmetic capabilities, poor (or at least poor man's) variables, and so on. However, Knuth himself has written that he would have used a better language if one had been available. Now that "we" are inserting Lua into the TeX executable it appears that this alternative has been identified (ironically, that link is about PyTeX, based on Python).

Now that \directlua is feasible, I don't see why it is not better in every way simply to make all "function" macros (those that do not directly store text) hook into that one primitive and then do all their programming in Lua. Loops, variables, recursion, and so on can all be done in a comprehensible way. A lot of effort in the development of LaTeX3 appears to be in managing the programming language and its use. So, I ask again:

Why use LaTeX3 to program at all? Why not turn its development into the development of a LaTeX-style document design library for LuaTeX, written in Lua?

This suggests a related question to me:

Is the separation of LuaTeX and LaTeX3 a result (or at least an artifact) of the non-communication among developers that Ahmed Musa described in his comment to this answer? What kind of cooperation is there between these two projects to reduce duplication of effort?

(An even more fantastical notion is to implement every primitive, except \directlua itself, in terms of Lua and various internal typesetting parameters, thus completely divorcing the programming side of TeX from the typesetting side. Then TeX itself would be a "macro package" on top of LuaTeX and one would not be bound to its strange design choices. An answer to why that should or should not be done might be too long for me to request, however.)

I don't personally think that this is too bloggy for this site, and I hope others agree. At least, given the confluence of expertise here, it seems to be the best place to ask the question.

---

I have accepted Frank Mittelbach's answer, though all the answers are extremely informative, because I think it hits all the main points most directly. One thing that neither it nor Joseph Wright's answer quite manages to come out and say directly is that ConTeXt does do the Lua-programming thing at least to some extent, and it does not solve all algorithmic problems because LuaTeX does not actually rewrite the underpinnings of TeX so much as stitch in a Lua layer that can interact with them.

I guess the one-line answer to my question is that my expectations of LuaTeX are much more grand than its reality, much like my "fantastical" idea.

Answer 1

(I guess I could be called a member of one of the teams ;-) this is my view)

I thought of staying out of this debate, but perhaps some words of clarification or, let's say, some thoughts are in order after all.

LaTeX3 versus pure Lua

First of all this is the wrong question imho: LaTeX3 has different goals to LuaTeX and those goals may well be still a defunct pipe dream, but if so they are unlikely to be resolved by pure Lua either.

So if one wants to develop an argument along those lines then it should be more like "Why does LaTeX3 use an underlying programming language based on eTeX and not on LuaTeX where a lot of functionality would be available in a "simpler" way?"

But LaTeX3 is really about three to four different levels

• underlying engine
• programming level
• typesetting element layer
• designer interface foundation layer
• document representation layer

See for example my talk at TUG 2011: http://www.latex-project.org/papers/

Here is a sketch of the architecture structure:

The chosen underlying engine (at the moment is any TeX engine with e-TeX extension). The programming level is what we call "expl3" and that is what I guess you are referring to if you say "LaTeX3" (and I sometimes do that too). However, it is only the bottom box in the above diagram. The more interesting parts are those that are above the programming level (and largely a pipe dream but moving nicely along now that the foundation on the programming level is stable). And for this part there is no comparison against Lua.

Why use LaTeX3 programming over Lua, when LuaTeX is available?

To build the upper layers it is extremely important to have a stable underlying base. As @egreg mentioned in chat: compare the package situation in 2.09 to the package situation in 2e. The moment there were standard protocols to build and interface packages the productivity increased a lot. However, the underlying programming level in 2e was and is still a mess which made a lot of things very complicated and often impossible to do in a reliable manner. Thus the need for a better underlying programming layer.

However, that programming layer is build on eTeX not because eTeX is the superior engine (it is compared to TeX but not with respect to other extensions, be it Lua or some other engine) but because it is a stable base available everywhere.

So eTeX + expl3 is a programming layer that the LaTeX3 team can rely on of not being further developed and changed (other than by us). It is also a base that is immediately available to everybody in the TeX world with the same level of functionality as all engines in use are implementing eTeX extensions.

Any larger level of modifications/improvements in the underlying engine is a hindrance to build the upper layers. True, some things may not work and some things may be more complicated to solve but the tasks we are looking at (well I am) the majority are very much independent of that layer anyway.

To make a few examples:

• good algorithms for automatically doing complex page layout aren't there (as algorithms) so Lua will not help you here unless somebody comes up with such algorithms first.

• something like "coffins", is thinking about approaching "design" and the importance here is how to think about it, not how to implement it (that comes second) -- see Is there no easier way to float objects and set margins? or LaTeX3 and pauper's coffins for examples

Having said that, the moment LuaTeX would be stable similar to eTeX (or a subset of LuaTeX at least) there might well good reasons for replacing the underlying engine and the program layer implementation. But it is not the focus (for now).

Why use LaTeX3 to program at all? Why not turn its development into the development of a LaTeX-style document design library for LuaTeX, written in Lua?

Could happen. But only if that "LuaTeX" would no longer be a moving target (because LaTeX3 on top would be moving target enough).

Side remark: @PatrickGundlach in his answer speculated that this answer that the LaTeX3 goal is backwards compatibility. Wrong. The same people that are coming down on you very strong about compatibility for LaTeX2e have a different mindset here. We do not believe that the interesting open questions that couldn't get resolved for 2e could be resolved in any form or shape with LaTeX3 in a document-compatible manner.

Input compatible: probably. But output compatibility for old documents, no chance if you want to get anything right.

But in any case, this is not an argument for or against implementing the ideas we are working on one day with a LuaTeX engine.

Is the separation of LuaTeX and LaTeX3 a result (or at least an artifact) of the non-communication among developers that Ahmed Musa described in his comment to this answer? What kind of cooperation is there between these two projects to reduce duplication of effort?

As I tried to explain, there is not much overlap in the first place. There is much more overlap in conceptual ideas on the level ConTeXt viz. LaTeX.

An even more fantastical notion is to implement every primitive, except \directlua itself, in terms of Lua and various internal typesetting parameters, thus completely divorcing the programming side of TeX from the typesetting side.

That brings us to a completely different level of discussion, namely is based on LuaTeX, or anything else for that matter, a completely different approach to a typesetting engine possible? That is a very interesting thought, but as @Patrick explained it isn't done with leaving TeX to do the typesetting and do everything else in a different language. So far such concepts have failed whether it was NTS or anything else because fundamentally (in my believe) we haven't yet grasped how to come up with a successful and different model for the TeX macro approach (as ugly as it might look in places).

Answer 2

Both Patrick and Frank have provided good answers: I'd like to add a few complementary points which seem too long for comments.

One aspect of expl3 that is important is that it can be used now to program LaTeX2e packages (the 'l3in2e' approach). That is seen for example in my own siunitx package, which I would not want to make LuaTeX-only. A particularly important aspect to this is xparse, which is proving to be popular with a reasonable number of users who probably don't care about the programming internals. This use of LaTeX3 code is dependent on being usable with a good range of the currently-available engines.

A second aspect is that LuaTeX does not mean you need no TeX programming at all. TeX is still there doing the typesetting, but also things like argument grabbing. (On the later, I asked the ConTeXt people about this once as I wondered about the old verbatim-in-arguments issue.) So improved TeX programming is still a good idea, even if you decided to go LuaTeX-only. Now, the nature of the structures you need would doubtless be different in that case, but again you can't simply ignore it.

Finally, there are practical considerations. A lot of the work on expl3 was done well before LuaTeX began, or certainly before it started reaching a more stable state. So to move from expl3 to Lua now would not really cut down on the work we need to do today, which as Frank says is mainly about higher-level issues. A lot of what needs to be done seems to me to be relatively insensitive to the choice of language: it seems unlikely that a switch to Lua would suddenly accelerate development.

Answer 3

(I am not a member of either team, this is my view.)

LaTeX3 is not only about providing a better output routine, but also about providing programming tools such as the famous quicksort algorithm (see l3prg.sty). This is obviously nicer to write in Lua (for most of us). Or consider string manipulation / regular expressions. This could be handled in Lua as well. But strictly speaking, one doesn't need Lua for that.

Why use LaTeX3 to program at all? Why not turn its development into the development of a LaTeX-style document design library for LuaTeX, written in Lua?

As I understand LaTeX3, one of its major goals is keeping backwards compatibility. While other documentation approaches are surely much appreciated in todays world (XML typesetting with CSS formatting for example), they would break existing documents. This is not what the LaTeX3 team wants.

What kind of cooperation is there between these two projects to reduce duplication of effort?

I can't see much communication between the two projects. LaTeX3 is aimed at "working with all engines that have the eTeX extensions". This implies that no engine specific code must be used (that is: functionality that behaves differently under different engines). Therefore there is no obvious need for LuaTeX specific code.

An even more fantastical notion is to implement every primitive, except \directlua itself, in terms of Lua and various internal typesetting parameters, thus completely divorcing the programming side of TeX from the typesetting side.

Lua does not do any macro expansion / replacement, you need TeX for that. It is not possible to delegate all work to Lua, you will need the TeX part of LuaTeX.

On the other hand, both LaTeX and Lua offer control structures such as loops or if/then/else clauses (LaTeX3 even more). Therefore you don't need Lua at all. If you mix control structures from Lua and TeX, the programming can get ugly, because you need to switch back and forth between TeX and Lua. Currently this is only possible with coroutines but may be easier in the future.

LuaTeX's great power in LaTeX lies in the manipulation of the (pre) typeset material.

Answer 4

My comments are from a user and programming perspective, as I am not a member of either Team.

The segmentation of the problem as Frank described it hit the nail on the head and should be the starting point of any critical review and comparison. To this we need to ask the question:

What problem are we solving and how can we summarize it?

Perhaps the answer is as simple as: "from a a set of markup typeset a document, based on a Document Description Language".

From this simple statement one can then start subdividing the problem into smaller parts, both from a user perspective as well as from a programming and algorithmic point of view.

So how are we to mark a document?

The obvious answer is, we need to type the text as text and the markup we need to denote it somehow different. I don't share the view that this can be simplified and shorter than as for example \makeindex even if users long for Markdown or Pandoc (good attempt to standardize markup), but so far this has been proven inadequate for complex commands or layouts. (all)TeX is flexible enough to adapt to such a change, if for example ten years from now markdown has evolved to marking a make index command to >makeindex and it produces a sitemap (if used on the web) or an index if used with an (all)TeX engine, that is fine, this can be adequately mapped. However, markup as is cannot compete with XML, which can represent a document more accurately even if LaTeX's environments come close. As XML is not for human consumption this issue needs thought, as it affects other areas of programming.

What happens next?

Simplistically the document with its markup is then passed onto a set of macros or another program for example in a browser which does a bit of processing, before passing it onto the next stage. With LaTeX or XML one can say we have a "Document Description Language" as compared to "Page Description Languages", such as PDF or raw TeX. Structuring documents this way is good and enables separation of presentation from structure and content.

LaTeX and others offers means to split this via the use of templates (i.e., what we call a class plus everything between the class and the \begin{document} command.

So far so good, but we got stuck on the Template Description Language that got stuck on programming concept fundamentals and algorithms and here neither LuaTeX or the LaTeX3 projects have offered anything substantially new. We also have limitations in touching the output routines which are currently over complicated.

As part of such a "Template Description Language", I attempted to have an ."ini" file where one would have specified settings for a template in a similar fashion to setting up an Apache or php settings file. Here LuaTeX was very good for parsing such a file (See Parsing files through LuaTeX).

Programming Constructs and Data Structures

Although I believe that (all)TeX by now has provided an adequate number of of data structures and constructs, either through the kernels or through packages, the work done by both the LuaTeX Team and LaTeX3 projects can ease the pain. Personally I would like to stay as far as possible within one programming paradigm and would prefer to stay with LaTeX3 even given it's alien syntax. On lower levels, such as font handling, encoding schemes etc. LuaTeX can be more useful.

So where to?

My general gripe with anything TeX, is the lack of agile development and a natural evolution towards goals, compensated to a great extend by the community and the rate of introduction of new packages. By natural evolution, I mean responding to user needs and providing solutions. The Teams need to address all level of users the author, template developer and programmer and so far somehow we got stuck on the user that is predominantly a programmer. It's fine with me as I tend to like puzzle solving;)

Answer 5

If I had my way, I would have loved a completely new system, with a well-thought out author approach, template designer approach, and programmer approach. Importantly, any new system must be endorsed by everyone as the way forward, and the new system must be able to do anything that users can do with LaTeX today. that is, it needs access to all the typesetting power and features of existing LaTeX and CTAN.

in its first incarnation, it should have compiled into LaTeX itself. We have enough processing power nowadays to live with such an extra-stage implementation. Eventually, as decades go by, the guts would be replaced and the TeX internals itself would fade into history, replaced by a more direct typesetting engine.

Alas, it wasn't to be...

Answer 6

Now you have another choice: the functional package. The package provides LaTeX2 interface for LaTeX3 programming layer (expl3) but emulates functional programming paradigm in Lua language:

• Every function has its return value (may be empty).
• The evaluation of composite functions are from inside to outside so it is quite easy to debug code.
• You can make every function and conditional become a group (like a block in Lua) so that local variables will be reset after the group.

Here is a simple example showing the similarity between lua code and functional code:

-- Lua code for comparison --
-- define a function --
function MathSquare (arg)
  local lTmpaInt = arg * arg
  return lTmpaInt
end
-- use the function --
print(MathSquare(5))
print(MathSquare(MathSquare(5)))

% -- functional code for comparison --
\IgnoreSpacesOn
\PrgNewFunction \MathSquare { m } {
  \IntSet \lTmpaInt { \IntEval {#1 * #1} }
  \Result { \Value \lTmpaInt }
}
\IgnoreSpacesOff
\MathSquare{5}
\MathSquare{\MathSquare{5}}

Note that with this package, you pass return values of functions with \Result command.