People are stochastic. You build reliable processes out of unreliable parts with feedback and self-correcting mechanisms. AI is not actually magically special in this regard. It has higher variance and we're still figuring out how to get all the tradeoffs right.
The big problem is that a person making a mistake can be taught to not make that mistake again. That's also not foolproof but at least it works a lot of the times. AI are unteachable, if you have given them a good prompt and they do something wrong 90% of the time you are shit out of luck.
That is to say I do agree that building reliable processes out of unreliable parts with feedback is the modus operandi. However AI cannot meaningfully handle feedback and learn. And that is a key unsolved problem.
> AI are unteachable, if you have given them a good prompt and they do something wrong 90% of the time you are shit out of luck.
If the Model makes repeated mistakes on the same subject matter, you can update your agent.md file, or you can add skills to deal with specific prompts, or you provide a better default harness.
The whole idea of coding agents is their harness makes a big difference vs a pure raw model.
> However AI cannot meaningfully handle feedback and learn
How do you think models are created? They are trained on feedback and learn.
Its not cheap but you can post train models. This is how custom models are mode, that deal with specific tasks more efficiently and accurately.
Example ... Composer? Its base Kimi v2.5 model that has been post-trained 2 weeks, to create Composer 2.5, what is a much better coding model.
Its literally trained to make less mistakes by feeding it correct data. Hell, a lot of the models you are using, are often the same base model, where v2.0 was the initial released model but the model keeps training, so when they release v2.1, its still the same model, but with more training time on feedback provided to v2.0.
LLM Models are not a cake you cook one time and they are done, and you start from zero again. If you have the money, and a powerful server setup, you can take a model like GLM 5.2 and post-train it, to reduce specific errors. Sure, you need a ton of money because its a large model.
But people have been doing this with 5M, 100M, 1B, 5B models for a long time already. To the point that some of the small models can do specific tasks, almost or better then some of the huge more general trained models.
> If the Model makes repeated mistakes on the same subject matter, you can update your agent.md file ...
That's all just prompting.
> How do you think models are created? They are trained on feedback and learn.
No one is post training models on a single mistake. At least I have not seen it. I also doubt it is effective. Post-training on a single failure will not meaningfully change the model. That even sidesteps the entire problem that you don't even have access to models if you use a provider like anthropic/openai
Teaching modifies the learner. Prompting doesn't modify the model. It provides additional context that influences a single inference. A person who has learned something can apply it years later without being reminded. An LLM generally cannot unless the knowledge is incorporated into the model itself or provided again.
I don't think it's generally thought of an advantage that LLMs are stateless. In deployment it's nice, it would be much better if the model could continually learn.
I'm not claiming LLMs are not useful, they most certainly are.
sleeping helps learning, but people can learn without sleep. You can teach something to a person and they can usually do it immediately in some capacity. If you prevent a person from sleeping they will have reduced capacity to remember things. But it will most certainly not be zero.
I don't know of any modern workflows that rely on "we'll tell the person not to do it again", though. There's a reason that companies have adopted blameless postmortems, because if your response to the DB going down is "It's fine, Kevin learns and next time he won't misuse the prod credentials", you are guaranteeing prod will go down again in the same way at some point.
Literally the entirety of the worlds infrastructure relies on that. In the past we had (literally) had nuclear war hinging on a single person just deciding that some data point is an artifact.
Every modern workflow implicitly relies on that. No infrastructure is fully robust. There's a senior DB person who has learned many things many times over who could bring down most of the US power grid.
> The big problem is that a person making a mistake can be taught to not make that mistake again. That's also not foolproof but at least it works a lot of the times. AI are unteachable, if you have given them a good prompt and they do something wrong 90% of the time you are shit out of luck.
I feel like this line of thinking is kind of an unfair comparison. I'm not saying LLMs are magical beings that can suddenly learn by themselves after getting something wrong, but your "person making mistake then being corrected" assumes you do tell the person about the mistake and tell them to avoid doing the same mistake in the future, but for the "LLM making mistake" example you then intentionally avoid letting the prompt being changed in response to the mistake, which would be the "then being corrected" part on the LLM side of the comparison.
Similarly, if you just let a person make a mistake and don't let them know about the mistake, they might keep making that same mistake over and over again.
If you update how you use the LLM as you discover what mistakes it does, just like you'd correct a person, then you can use an LLM and also the LLM can "be taught to not make that mistake again".
I'm not against the prompt being changed, the point I was making is that an LLM is prone to the exact same mistakes even if you change the prompt. A trivial example is the very basic character counting mistake, I just asked chatgpt:
> How many p are in strawperry?
> There are 0 “p”s in strawperry.
And I can trigger the same mistake with various words even when adjusting the prompt many times. So I cannot teach chatgpt to correctly count characters.
> the point I was making is that an LLM is prone to the exact same mistakes even if you change the prompt
That'd be strange, what specific models?
Your example of "changing the prompt" doesn't actually tell the model what to avoid though, you're just rephrasing the question? Again you're not doing an equal comparison for the human, because in that example I bet you'd tell the human what they messed up and how to do it better, why not include that in the prompt when you compare these two?
I have a clojure/babashka environment that I work in. My agents/claude files specifically explain this and how to work with it and what to use etc. Fable decided it didn't want to use bb/clj and ran python to do cli work in one project.
So I update the files in all projects to specify that python is strictly off limits/not to be used/never to run it/etc and created a preToolUse hook -- because I learned that just because an agents file says something doesn't mean it gets followed. Hence the hooks, fool me once ...
Later, I start work in another project with the hooks and updated files saying to use the established bb/cli tools and to absolutely never run/use python etc. What does it do on the first task? Tries to run python, gets hit with the preToolUse hook error and replies with something like -- oops my bad I tried to use python even though the instructions told me not too. Every once in awhile in new tasks I still see the hook error at some point but it doesn't apologize like it did before it just switches to proper tools after it can't run what it wants even though it's in the instructions and could have been avoided.
"AI are unteachable, if you have given them a good prompt and they do something wrong 90% of the time you are shit out of luck."
please take a look at the error(s) made in the prior run. what could've been done better? create or modify an existing skill to emphasize this, or suggest additional language in AGENTS.md.
It will return a bunch of relevant-sounding insight, modify skills and context files… Then do the same error again.
We’re not at the point where AI is capable of knowing what went wrong and self-aware enough to understand how it could reliably change its own behavior.
For months I’ve been trying to have the agents stop manually writing our auto-generated SQL migrations and run the command that generates them instead. SOTA models insist on occasionally getting it wrong.
Indeed. Any meaningful AGI/ASI will have to have a form of memory / continual learning. Sam Altman said last year that this will be the focus for GPT-6.
The whole "soul.md" stuff today is a poor approximation to that. But I wonder whether it will grow into it, like chain of thought prompting grew into reasoning models.
LLM's as a technology - currently - are stateless. The memory layer is controlled by the agent. I m surprised with the gpt-6 reference unless it has to do with vertical integration between the agent and the layer.
> However AI cannot meaningfully handle feedback and learn.
Well this is the central bet of AI coding isn't it? We, the humans-in-the-loop, get better at knowing ahead of time which patterns AI will handle better than others, all the while the models actually get better.
> The big problem is that a person making a mistake can be taught to not make that mistake again. That's also not foolproof but at least it works a lot of the times.
You can do the same to an LLM with the same outcome.
People live in very stochastic and volatile environments and they manage that in ways no LLMs currently ever can. (ie: imagine sending an LLM all the data - sensory/auditory/etc… - that a human receive)
People’s job is to partially reign in this volatile environment by creating processes with stable output.
Please. If you told a customer support rep that you are the former US president [0], they would not hand over the account straight away because you asked nicely.
These models are great tools, but putting them and people on the same level does a disservice to our species and also is simply incorrect to what we know these models to be and their capabilities/limitations.
Most tasks we use computers for are deterministic and was coded for that specific quality. Introducing nom deterministic behavior is lowering the value of the app, especially for power users.
I don't think this is true. Computers are still computers, and code executes (mostly) deterministically. What computers struggle with is making sense of unclear inputs, precisely because of their determinism. OCR, speech-to-text, and computer vision all proved extremely limited when tackled purely deterministically. The whole "ML revolution" before LLMs was about recognizing that messy, noisy data can be interpreted with some accuracy using statistical methods. LLMs are the continuation of that: they are even harder to measure and completely impossible to prove, but they bring capabilities we have been unable to achieve any other way for decades. Should they be used to handle deterministic logic? No, obviously not - but they do enable computer systems to start working in contexts where they either couldn't at all before, or were just bad at them.
