The problem is that at the business level and at least for a large part of the US government, we need to assume that there are no 'good' actors.
Anthropic needs regulation in order to prevent AI from becoming a commodity. This of course does not benefit all tech businesses equally, especially those that are not currently at the AI frontier. So when JD Vance talks about AI, he talks using the mouth of Peter Thiel who may not see benefit from the same policy as Altman or Amodei.
The rest is just public support posturing and most of that is bullshit meant to distract from the high rollers game of winners and losers. The philosophy is money and power, who gets it and who doesn't. Us normies aren't really participants in the game, except where we are being manipulated into cheering for one side or another, and with little stake in the outcomes (although selfishly, I'd be pissed if I didn't have open models to tinker with).
Open source is fundamentally a vehicle for commoditization. This is great if your business is not AI and your business is instead something like GPU hardware or some product that uses AI. But it means that eventually, selling AI is not going to be the money maker.
OSI proliferated open source on a business strategy called "commoditizing your complements". These big companies don't do it out of benevolence. It was pitched to them in a way that FSF did not (which was more about morals and ethics, something business care little about), and it caught on. And the software business became about ads, consulting and cloud services instead.
It's also clearly the wrong question for this decade. Private / public sector cooperation and all of the barriers between them that have been degraded and chipped away has left us with a very broken economic system that looks less and less like capitalism every day.
One only needs to read Stevens' dissent in hindsight to recognize that all of his concerns were and are legitimate, and the worst of which have come to pass.
Around 2015, my boss got permission from corporate to take me and a few of my coworkers to a leadership conference for a day. It was setup at a church and had religious theming around it, and a lot of "leadership values" were communicated. It was the usual fare of responsibility, accountability, truth and trust you'd expect, mixed with the religious overtones you'd also expect. But what I didn't expect is how trite this fictional narrative would prove to be.
I think about that a lot now, and how my religious upbringing was bombarded with this messaging (in a good way). And how much of the same world that previously claimed espouse it has complete turned their backs on it in such a short time.
Is it dead or did it ever exist at all? At least we don't seem to pretend to have higher values anymore. If we even have leaders at all, they are not measured by the yardsticks we used to make.
If you look at papers on benchmarks, they're usually created to expose gaps in how models are trained. It should be no surprise that models get better on them over time, because you can't get better at what you don't measure.
Cherry picking the benchmarks you present is where the falsehoods lie.
Another thing that sort of puzzles me about benchmarks is that LLMs are not deterministic and do not always complete a problem. So what are the results actually representing? The best run? The average? It is all in some ways a falsehood
That's a good point and conventionally if benchmarks aren't run as "one shot", it is denoted as "benchmark@K". Inference time scaling has historically shown improvement.
Generally though, many of these fairness complaints do go away if there is "3rd party testing". Right now, companies reporting their own benchmarks has all the problems that 3rd party testing resolves in many other industries.
reply