This is too long for a comment, so I'll post it as an answer:
Your points in favor and against TeX
- math […] is unparalleled, but it's already ported as MathJax
That's not quite true. MathJax copied a part of the syntax, but in direct comparison, there are quite a number of differences in the output and MathJax cannot compete with the typesetting system TeX. But of course, for most users both systems output some math notation they can understand. This point is definitely about the quality you want to achieve.
- offline processing, which enables much more careful calculations of typesetting details like spacing and hyphenations comparing to (online) browsers and (WYSIWYG) MS-Word; […] should be possible in headless browsers.
Now the question is what you want to emulate. Even with headless browsers a careful layout of pages (as TeX does) is hard to emulate because it's quite computation-heavy. Okay, the hyphenation algorithm is already implemented in browsers (iirc) and you will most likely even get some kind of page layout (pdf.js also uses web technology to output pages). But will you be able to make detailed font-specific adjustments? Probably not without a very large stack of additional components.
You should be very careful that at this point calculations like the one pgfplots does and those made by the engine do not mix and you do not end up with a kind of resource monster that will eat up more memory than my browser window with 100+ tabs (btw: I would expect a modern web application to run in 1 tab and not to eat too much resources, that will be a big challenge for you, I guess).
- a uniform macro language both for controlling and for text generation; […] might be an advantage decades ago when Unicode was not popular or even not born. Now people use commands almost only for non-texts like figures and code listing. When special text is constantly used, as \TeX in texbook, it's easy to emulate in html+css+js anyway
With that you are quite fast. Do you really think that unicode solves all problems? I understand that the web does use different techniques for semantic markup (that's what macros are also used for, you refer to figures and listings). But all the advantages TeX offers are based on macros as a general purpose language for typesetting in this case.
Modern applications use similar approaches: domain specific languages. TeX has formed a DSL, one that is nowadays adopted by others (example: MathJax), but please note that your "easy to emulate" might not be as easy as you think. What is important is the pleasure of working with macro expansion that solves many problems and doing macro expansion as "funny excercise" will never achieve a way to emulate TeX in the browser.
Going back to the processing point of view: TeX integrates many options into its process, including the use of external tools like Biber, Xindy etc. If you want to port the whole user experience (i.e. the output) you would need to port these pieces of software as well.
for LaTeX and the like, there is separation between content and style. […] is now much more discussed and supported in the front-end community.
That would be a point for you. HTML and CSS are much better in that discipline than TeX and LaTeX are. But really if you attempt to achieve a separation you are able to get one.
Your aim
My motivation of porting *TeX comes from the countless frustrating hours of futile efforts of debugging and googling and ordering packages and so on.
So here we go: porting TeX. There have been some projects of porting TeX to type-safe languages (okay, Java) in the past, namely NTS and ExTeX. I have to confess, I had a look at the source code of both in the past and even experimented with them to run simple TeX files, but as complete TeX implementations they have one aim: Running TeX files the TeX way – and that means with expansion.
What you are trying to do is to write TeX without TeX and that's basically what the LuaTeX team offers with Lua as a separate language. But you want something else: Remove TeX's intention from TeX. You want to use TeX (its algorithms etc.) to produce web pages which are quite different from what TeX was made for, namely typesetting on paper (restricted size, printer restrictions etc.).
I am aware of LuaLaTeX. Even if it solves some of the problems had all the major packages been rewritten, it doesn't seem good enough to let me ignore the existing and future advancement of front-end standards and technologies.
What do you expect the community to do here? Most, if not all, of the package maintainers are trying to be engine-agnostic, so they can run on pdflatex, xelatex and lualatex with the same results. Optimization for one engine has to have real benefits (see e.g. pgfplots), otherwise you won't do that (Knuth: Premature optimization is the root of all evil).
And there is another important fact: Most, if not all, package maintainers do not get any money for their work, they do that voluntarily. If you'd reimplement TeX (and discard the macro approach) you would not be able to just say "Hey, I got a new 'engine', please adjust your packages", because the TeX community loves its open-source software which has proven to work.
Therefore, I would like to ask if I missed any significant advantage of *TeX that is hard to be ported or any existing effors for making *TeX modern or useable enough, or should I ask somewhere else?
So basically I think you missed one important advantage of TeX: It's good in (most of) what it does, as a whole. It does very good typesetting and with ConTeXt and the more recent engines, you even have really modern additions available. All of that aims at one thing: output that is sliced into pages, be it frames of a presentation or pages of paper. Really, it is too easy to say: "That, that and that is good and I dislike that, that and that, so let's just port the former and remove/ignore the latter".
And one more: Making TeX modern and usable currently is a process of a whole community (projects like expl3, LuaLaTeX, PDF tagging, font handling, …), so it will be hard to really outscore all of this with a single new project.
Answering your comments
(There are quite a number of differences in the output and MathJax cannot compete with the typesetting system TeX.) Could you please elaborate on this?
MathJax is quite good at transforming a mathematical input notation into more appealing math output. What it does not do (as far as I know, please correct me if I'm wrong) is the whole computation part for positioning TeX does (which Moriambar explains in posts like this one).
Apropos TeX support: They have quite good FAQs showing that they do not intend to replicate TeX (e.g. http://docs.mathjax.org/en/latest/misc/faq.html#does-mathjax-support-tex-macros, where they only refer to LaTeX).
What I think is more important here is the point that while MathJax produces "good" output, it is by far not as extensible as TeX is. Think of the ton of math packages out there, which are not always portable to MathJax. That may be because of font issues or due to TeXs programming capabilities which cannot easily be transferred to MathJax (think of the many options you have with expl3 which are lost there).
If you really want to build a TeX clone with web technologies I suggest one thing: Search for questions on this site which are tagged with math-mode and which contain complex examples of math issues which have been solved by our power users solely with TeX magic. If your clone enables this too, everything is fine, otherwise one probably has to know many languages (TeX syntax, HTML, CSS, JS) to achieve similar results in your system which would be much effort for small reward.
It's counter-intuitive to me because the program part of the whole tex.web contains only a few thousands SLOC therefore it's math subset should be easy to clone.
Okay, you measure TeX based on the lines of code? Then here's a warning: Donald Knuth has implemented many optimizations. I guess you can easily clone this if you have understood the whole part on math typesetting (and everything connected: fonts, alignments, …) in the TeX book and also what the code really does. But otherwise, it might not be that easy. As I said, this statement only holds true if you intend to replicate TeX. If you do not want (nearly) identical output, then you are still fine using whatever code you are able to come up with.
(Will you be able to make detailed font-specific adjustments?) I'm indeed unfamiliar with manipulating fonts. Could you give me an example?
Yes: http://mirrors.ctan.org/macros/latex/contrib/microtype/microtype.pdf
(But please note that your "easy to emulate" might not be as easy as you think.) Could you give me a counter-example?
As I said, there have been reimplementations like NTS and ExTeX that tried to do it right. And they are quite complex. So maybe you have a look at their source code. This point is all about compatibility: If you want to "port TeX" you must care about that, if not then you are good to go with any part of the syntax.
But if you want to port TeX, have a look at packages like the LaTeX and LaTeX3 kernel and their expansion trickery, packages like TikZ which rely heavily on the output driver (a thing you probably had to write for the web) and if you want to compete (and be compatible with) with modern engines like LuaLaTeX you should have a look at their callback mechanism.
(What is important is the pleasure of working with macro expansion ...) I understand it's a pleasant toy to some people. I used to be fanatic in writing poems in Perl and writing programs in gnu-dc. But for some people they're not fun but unnecessarily painful instead.
Nobody argued that the learning curve would be an advantage of TeX. Most people say it's hard to learn and they may be right. It's definitely hard to understand TeX (personally I do not always understand it). So you want to remove macro expansion as one of the core concepts and still have a tool as powerful as TeX? Then you (a) need to introduce another language (possibly with a better learning curve) and (b) lose most TeX users who are used to the way TeX works and accepted that macro expansion is a concept (not the most comfortable, but not the worst either).
(But you want something else: Remove TeX's intention from TeX.) I doubt what Knuth's intention really were but if it's about "only expansion", then you're right.
I guess I did not make my point clear: Knuth's intention was to create a typesetting system for his books. Books (as far as I know) were collections of pages, something printed, at this time, so he never targeted digital output in the form we know it today. TeX is based on the assumption that there are pages, that the output routine has to calculate break positions etc.
(What do you expect the community to do here? ...) Sorry for the misunderstanding. I am certainly not blaming the community. I do feel that it's kind of repetitive work for unnecessary backward compatibility.
Okay, so you are arguing that backward compatibility is unnecessary. Then feel free to use systems like the one digital scratch paper by that big company everybody loves and happily accept that you need to fine-tune a document with every new version. Compatibility may seem like a "feature" blocking further progress, but really it is one of the advantages of TeX.
That said, it is certainly not repetitive work to adjust packages to engines like LuaLaTeX (that was your point). That is a conceptional decision and many packages do not need to adjust anything, because the modern engines decided to be compatible and work like TeX. Yes, it's true that you may miss some advances in front-end technology by being backward-compatible, but if I am up-to-date, PDF is still the way to go for paper-based digital exchange, so TeX is (in its category) using modern technology.
(It's good in (most of) what it does...) except when it comes to image positioning and code listing which cost me days for nothing. I no longer care if there is one specific way to write (or fix) the macros to achive what I wanted. They are obvious problems and deserve obvious answers or messages. The lack of them would be the definition of bug.
Yes, you are right and there is a new output routine for the LaTeX3 project (xor if I'm not mistaking). Floats are conceptually hard because in a paper-based workflow they have to fit on a page with other text that fills the page. That said you are free to do what you would do in other software: manual positioning (TeX lets you do that).
And yes, you are right, obvious problems deserve obvious answers, but what you want at this point is a feature-request not a bug fix, because any major change in the output routine destroys backward compatibility.
Last, but not least, I wish you luck. Really, I would love to see a modern competitor to TeX, but up to now, nobody has come up with anything that could really compete.
