I don't have any insight at that level of abstraction. Figure placement like a bunch of other things has required me to hack around to get my desired result. Is there any newer research on these algorithms that could make it better given much better hardware?
As for LuaTex I wasn't able to find any examples of actual markup written in it. I would expect that for the actual markup you'd want a language with a bigger and more flexible grammar so you could build DSLs. Ruby seems particularly good at that kind of stuff, but I'm biased there.
LuaTeX has the same macro language as the other TeX engines (pdftex and it's ilk). The true power of LuaTeX comes into play when you use Lua scripting. I am not sure if ConTeXt has more advanced figure placement algorithms due to LuaTeX. LaTeX for sure has the same algorithms as before.
But then is the Lua scripting at the user level? Can it be used in the building of documents? Or is it a scripting language for the underlying engine, invisible to the end-user?
Incidentally I went looking for what pure TeX actually looks like as most people say TeX when they really mean LaTeX. Here's Knuth's TAOCP Volume I errata:
I wrote a LaTeX macro package for my university's PhD thesis format, and like a lot of such packages, the bulk of the work was in TeX rather than LaTeX. It's...well, I'm struggling to find the appropriate word that means "awful, but sort of comfortably awful."
Like a lot of things, if you're handed a finished product, it looks like random noise. If you could watch it being written, it pretty much makes perfect sense. It's a hideous syntax, and sometimes I still wake up in a cold sweat with long strings of '#' signs running taunting me from the corner of my eyes. But I can't say it was all that daunting to write, although I suspect my TeX code was highly non-idiomatic.
You can do Lua scripting a the user level (\directlua{...}). But you have to know the Lua interface. What purpose is having Lua inside your document? Do some calculations? Yes, that's easy. Do complex algorithms for placing floats? Then you need to replace a lot of TeX/LaTeX code anyway and then I wouldn't speak of user level. What you can do easily from inside Lua: shuffle nodes (nodelists) around. Everything you see on a page is represented by a node. So you can reposition anything. That requires a lot of knowledge of the internal representation of nodes and the LuaTeX API, but once you grok it, it is much fun to do these things.
And no, Knuth is definitely not the only one using plain TeX. Yes, it is a niche.
I did a few times. It's actually easier to make something look exactly the way you want it to look in plain TeX than fight with some poorly documented LaTeX package.
I'm sure its possible and useful for specialty cases but would you want to write TAOCP in it?
My experience of using LaTeX compared to WYSIWYG is that it's easier to focus on the text in the editor instead of fighting with the UI interface but it then requires some chasing around package bugs and compile errors. In total it's clearly a net positive for me.
Writing pure TeX would seem to push the tradeoff quite a bit, having so much distracting stuff around the text that requires a totally different mindset than the text being written. It was probably designed for a world where typesetting was done as a final step before publication, potentially by someone else.
The idea is that you put together the few macros that you need and then the book is mostly tag-free.
IDK you can find the source to the TeXbook on CTAN, see if it seems too noisy to you (I don't remember).
As for LuaTex I wasn't able to find any examples of actual markup written in it. I would expect that for the actual markup you'd want a language with a bigger and more flexible grammar so you could build DSLs. Ruby seems particularly good at that kind of stuff, but I'm biased there.