It's a very misleading: they don't provide any meaningful documentation/requirements. Just an executable blackbox.
E.g. a doc for ffmpeg, which I checked by downloading docker image they provide to the model, is a README which basically just says this is ffmpeg and docs can be found online. They do not allow models to get online.
So a model is supposed to reverse-engineer a blackbox using only limited number of tries. I'm not sure even ASI can do this under these constraints (without memorizing the ffmpeg code base, obviously.)
In the only posts one of authors mentions "usage docs". Obviously they had a command-line tool like `grep` in mind -- where a man page sort-of specifies program behavior. But then added sqlite, ffmpeg, php, etc. - where a usage doc is like one millionth of information you need to implement ffmpeg.
And, of course, there's no human baseline. I'd guess making such a baseline would cost billions of dollars.
SMB is a rather basic RPC protocol with just a handful of different types of calls. It's many orders of magnitude simpler than audio/video compression formats.
I think you underestimate complexity of audio & video encoding standards. There are hundreds and hundreds of pages of specification. How many times do you need to execute real ffmpeg to get all tiny details?
It's certainly possible to reverse-engineer it from a blackbox access, but it would take *years* and this test has a time limit.
Even given that I think solving the problem would require a certain amount of personal agency and volition to drive useful experimentation, and then you still have an inescapable problem that a design process is never verifiably done; it just a sense of taste when a product is good enough and it’s time to stop working on it.
I’m not sure this benchmark is even very interesting because it requires a language model do something that it really cannot do. Maybe it would be possible with a novel harness in an ensemble system, but I would never expect a pure language model that is run in a minimal harness to ever be able to do this.
E.g. a doc for ffmpeg, which I checked by downloading docker image they provide to the model, is a README which basically just says this is ffmpeg and docs can be found online. They do not allow models to get online.
So a model is supposed to reverse-engineer a blackbox using only limited number of tries. I'm not sure even ASI can do this under these constraints (without memorizing the ffmpeg code base, obviously.)
In the only posts one of authors mentions "usage docs". Obviously they had a command-line tool like `grep` in mind -- where a man page sort-of specifies program behavior. But then added sqlite, ffmpeg, php, etc. - where a usage doc is like one millionth of information you need to implement ffmpeg.
And, of course, there's no human baseline. I'd guess making such a baseline would cost billions of dollars.