CircleMUD was my first intro to C and network programming. I was at a computer camp for 8-10 year olds that taught BASIC. One of the counsellors was doing their undergrad at caltech and pulled the students who were bored by BASIC into a separate group to learn C by editing CircleMUD.
Hacking up a pre-existing MUD was the perfect mix of immediacy / interactivity and fun to hook me. So thankful to that guy, wherever he is. He probably changed the course of my life.
I know nothing about the PhD candidate and professor who co-authored this, but I don't think this statement makes sense in general.
Typically CS labs at universities contain people who have not spent a significant amount of time exposed to large industry codebases and the corresponding complexity. I think the post would have more credibility coming from e.g. the platforms team at a tech company with a monorepo.
Well, they’re also probably not scaling up in the same way as any of the commercial AI offerings, let alone the frontier labs. Faculty at Stanford probably have some decent hardware to play with, but they do not have data centers. I don’t doubt that they know their way around CUDA/PTX, but it’s not clear how relevant their message is given that their research code is very likely not being deployed in production, or at scale.
agents are unlike humans in many ways but they have their own limitations. it's magical thinking to believe that they are able to cope with zero software abstractions, especially as a codebase increases in size and complexity
and today's agents increase codebase size and complexity much faster than humans do via traditional software engineering methods, so this is likely to hit a wall faster than if human software teams were to do this experiment.
Just yesterday, for fun, I asked Codex to generate an “online e-commerce site” in ARM assembly. To its credit, it did actually generate a program that served up a webpage over a socket, but it really did not do what I asked. The program still pulled in a large number of shared libraries (something I’ve never really thought to try in assembly). And… it was not actually a functioning e-commerce site. Iterating on that initial prompt proved painful—the program was extremely brittle and now I had to review and talk about ASSEMBLY. Maybe we can dispense with abstractions someday, but holy cow, not yet. I would much rather write an e-commerce platform in C than try to vibe-assembly my way to one.
Your work lets you burn infinite tokens on hobby projects? That's wild. My work has no token limits per se, but we definitely aren't supposed to spend tokens on side projects.
Don't make me tap the sign: There is no such thing as "bytes". There are only encodings. UTF-8 is the encoding most people are using when they talk about modeling "raw bytes" of text. UTF-8 is just a shitty (biased) human-designed tokenizer of the unicode codepoints.
Virtually all current tokenization schemes do work at the raw byte level, not the utf-8 character. They do this to avoid the Out of Vocabulary (OOV) or unknown token problem. In older models, if you came across something in the data you can't tokenize, you add a <UNK>. But tokenization should be exactly reversible, so now people use subword tokenizers including all 256 single bytes in the vocab. That way you can always represent any text by dropping down to the single byte level. The other alternative would be to add all utf-8 code points to the vocabulary, but there are more than 150k of those, and enough are rare, that many would be undertrained. You'd have a lot of glitch tokens (https://arxiv.org/abs/2405.05417). That does mean an LLM isn't 100% guaranteed to output well formed utf-8.
And in regard to utf-8 being a shitty biased tokenizer, here is recent paper trying to design a better style of encoding https://arxiv.org/abs/2505.24689
Roger, who spoke only Chinglish and never paused between words, was working on a VAX FORTRAN program that exchanged tapes with IBM mainframes and a memory mapped section, inventing a new word in the process that still has me rolling decades later: ebsah-dicky-asky-codah
RMSNorm is pretty insigificant in terms of the overall compute in a transformer though -- usually the reduction work can be fused with earlier or later operations.
Rmsnorm acts like a barrier. No compute on the next network layer can start before all compute in the previous layer is done.
Splitting networks across multiple GPU's, this means you must wait for the slowest node and the longest latency.
As soon as you can remove most of these barriers, compute over non-latency-guaranteed networks becomes more practical, as does non-homogeneous compute (ie. Mixing different GPU models).
Yes of course it can, because they fit in the context window. But this is an awful test of the model's capabilities because it was certainly trained on these books and websites talking about the books and the HP universe.
Given that it is pretrained on the material, it would be interesting to do a differential test on in-context reinforcement. What is the recall % before reading the books and after?
I know, for instance, that gpt4 does much better with the python manual when we quote relevant context, even though it was trained on the python manual. This suggests pretraining is less than perfect.
Likewise, in the Harry Potter case I expect a significant difference between its background knowledge and the context enhanced trial. But I don't have intuition about the effect size we should expect! That makes it a fun experiment.
Not so fast. If you were evaluating the model on its ability to predict the next word in a Harry Potter book, you'd be right, because it's already seen the entire book, but that's not what's happening here.
The linked X post shows that the user asked the model to generate a graph of the characters, which was presumably a novel question. This is a legitimate test of the model's ability to understand and answer questions about the training data. Repeating the books in the prompt for emphasis makes sense, since the model probably didn't memorize all the relevant details.
The training data may not be HP itself. It may be millions of pages summarising/discussing/dissecting HP, which already contain the relationships spelled out better than in the book itself.
That's true, but the model still analyzed all that disparate information and produced a very detailed graph of the relevant relationships. If anyone can show that the graph itself was in the training data, then I would agree that it's not a good test.
The frustrating thing about all this speculations is, that we don't know what was in the training data, but I think we should know that, to have any meaningful discussion about it.
It's a novel question and impressive that Gemini was able to solve it but the tweet's author is claiming that this is because of the large context window and not because of all the Harry Potter related training data that was available to it.
The generic models definitely know a lot about Harry Potter without any additional context.
Probably 80% of my questions to ChatGPT were about Harry Potter plot and character details as my kid was reading the books. It was extremely knowledgeable about all the minutiae, probably thanks to all the online discussion more than the books themselves. It was actually the first LLM killer app for me.
That's a good point. I would describe this as a test of Gemini's ability to re-read something it's already familiar with, not a valid test of its ability to read a large new corpus.
It could have been trained on this exact picture created by a fan and uploaded to some forum. Ultimately it is impossible to know unless testing with brand new material.
I have the same problem with benchmarks that use real world tests (like SAT/LSAT/GRE or whatever else). The model got a good score, sure, but how many thousands of variations of this exact test was it trained on? How many questions did it encounter that were similar or the same?
It seems from the replies that he tried it without the context too and didn't get as detailed answers. I'd really like to see the actual difference, but yeah, it would be so much more interesting to use books which aren't summarised and discussed all over internet.
Hacking up a pre-existing MUD was the perfect mix of immediacy / interactivity and fun to hook me. So thankful to that guy, wherever he is. He probably changed the course of my life.