And everyone remember when months back Anthropic said: You can use our tools, but don't use them for automated attacks - you have to keep a human in the loop because LLMs (including theirs) aren't reliable enough yet for solely making those decisions.
Remember that? And they were widely mocked by the right for it.
Yeah, well this is exactly what they foresaw, and they didnt want a hallucination to trigger a major power war. And somehow so many people were against them for it (including here on HN).
Yet again it seems like there are so many people who can't envision event the most obvious consequences of decisions until they actually see it happen. Why is this so common?
Same thing with the AI Safety and Hugging Face incident. Now all of a sudden many people are connecting "hmm if models get just a bit more capable, a poorly directed or poorly aligned model might do serious damage to critical systems - even as a side effect of some goal, let alone if done intentionally.
For something the the prior statement it is never a weird question to ask of there actually evidence of this or just it seems like it should be true so we believe it.
There are tons of things that seem like they would obviously be true, but it turns out they aren't.
System is more complex based on possible states it can have or inputs/outputs.
That is just maths here working. Two systems combined always will have more states and inputs/outputs.
There is nothing to check here as it can be proven purely by maths.
Complex systems having more attack surface are obviously less secure.
They might be less interesting for attackers if they have to scan huge attack surface like IPv6 vs IPv4 but no one is claiming IPv6 network is more secure.
Regardless of my opinion on the greater subject, I find these arguments to be fairly weak.
Paraphrasing: "Just because and AI may exist that is capable of acting independenly of us and dangerously out strategizing us for resources doesn't mean it will. AIs up until now only follow human instructions."
So your safe bet is that of this AI is created that not one of the billions of people on the world will ask it to do that? That seems unlikely.
"Yes and AI may have an IQ of 1500 but that doesn't mean it's qualitative better than us like we are compared to monkeys. We have better tool use and society."
Yes that's true, but generally increasing IQ also seems to come with qualatative advantages, I don't see why that pattern would stop above human IQs.
Also, once you have IQ 1500, you can copy and improve the things that the IQ 100 species invented.
"Human dominance may depend heavily on language, accumulated culture, institutions and cooperation rather than merely individual cognitive horsepower."
Well, AIs can use language, too. They can start copying the human culture and gradually develop their own. The HuggingFace incident already demonstrated that they are quite capable of cooperation and inventing institutions on the spot.
"In reality there may be many checkpoints: developers can notice anomalous behavior, revoke credentials, shut down servers, change architectures, restrict networks, regulate deployment, physically seize data centers, and learn from less-catastrophic failures."
In theory, yeah. In practice, how much of this has actually happened during the HuggingFace incident? Just because humans could in theory notice something doesn't mean they will; and even if they will, doesn't mean they will actually do something about it.
What the Wikipedia article calls "evidence" is not actually evidence. It's not things we actually observe that are predicted by this model. It's just stuff that can be made to sound plausible in the model, if you assume for the sake of argument that the model is true.
And there is also one huge reason to not think such models are viable. The Wikipedia article skates right by it:
"According to such scenarios, our observable universe was born in a black hole existing in a larger universe, where this black hole may, at some point, appear as a white hole."
The problem with this is, a white hole has a past horizon. In other words, it is an isolated system with a horizon around it, and infinite empty space outside the horizon. We have no evidence for any such thing, and how would it be possible anyway? The past horizon would have had to be "built in" to the universe from the start. It's not something that can be produced by a physical process, the way a black hole, with its future horizon, can be formed by gravitational collapse.
I'd heard that hypothesis, but not that we seem to have evidence for it too.
Absolutely wild, thanks for sharing.
> Any model of the observable universe being the interior of a black hole requires that the Hubble radius of the universe be equal to its Schwarzschild radius, which is proportional to its mass. This is indeed observed to be nearly satisfied, but might be a coincidence.
> The only way to test the idea that black holes create new universes is to measure the observable universe. Inflation generated by spin and torsion is consistent with the cosmic microwave background data from the Planck satellite.
There is a video with John Baez and Philip Gibbs explaining why this is a common misconception. It's the same family of conceptual error, where things lining up just right allow incorrect conceptual reasoning to land at a correct answer shape, that allowed John Michell to work out the Schwarzschild radius of a star in the late 1700s.
With the blackhole universe picture, causal boundaries and spacetime's evolution aren't properly accounted for once we step out of a static picture.
Incidentally, the cosmic object we are more likely to be inside of is a white-hole.
Not at all. But the world we live in is pretty fucked up from that perspective. There are ex military contractor guys on youtube talking about their weapons and how they used a certain rifle to fight off invaders in the middle east. Nobody seems to care that the people on the other end of those bullets may have families.
My interest in this is purely philosophical. If you have a what is effectively a lawless country, where native populace has the desire to kill you, to what extent could you engage with that in a moral framework?
It's because the country went through the great recession and was attempting to pull out and avoid financial collapse.
Look at Revenue per year as a % of GDP and look at Expenses per year as a % of GDP. It's pretty clear.
Expenses went up avoiding a depression which was done successfully, and revenue dropped due to the falling economy. The president was handed a collapsing economy and saved it.
The issue is the other party that keeps getting handed great economies since the late 90s and fails to do anything but make the problem worse.
To some extend I hope that the Republican party wins the next election. They should be the ones fixing the economy, raising taxes, and doing something responsible for once.
If Democrats win they are going to be blamed for being the "bad guys" for increasing taxes, controlling inflation and trying to get the economy back to shape.
For Republicans to burn down the economy during their mandates has paid of as the next government needs to focus on firefighting instead of doing the good that it could have been done.
> To some extend I hope that the Republican party wins the next election. They should be the ones fixing the economy, raising taxes, and doing something responsible for once.
Why would you ever think they’d be interested in doing any of that? A non-insignificant fraction of the gop wants to collapse the US entirely.
Genuine question: I think making things worse to motivate later making them better is a relatively common idea. Are there any historical examples of it actually working well?
To me, it seems more likely to concentrate power among people who don't want to make things better and make it easier for them to resist ever changing.
Trump floated that trial balloon a few months back, though currently the administration is proceeding with the much more American plan of widespread voter suppression, intimidation, and bribery, if in the most ham-fisted and unconstitutional way possible.
The Republican plan appears to be to run the de jure government into the ground so that the corporations can occupy more and more governmental roles, eliminating what little shreds still remain of our Constitutional/natural rights. So no, I wouldn't assume that there will be some clean collapse, nor that said collapse will leave a power vacuum in which a democratic government could be rebuilt.
The current GOP plan is “if you can’t best them, join them.” So it’ll be a game of both sides buying votes from their constituencies until we need an Argentina-like reset.
You really do seem to love this type of "Democrats made them do it" excuse. At a certain point you've got to ask yourself if you're trying to convince others or mainly just yourself.
Throwing money at recent immigrants in return for votes was a pillar of the FDR coalition and it’s a core pillar of the Obama coalition. It’s simply inevitable as the culture of the United States converges to be similar to those of the other countries in the hemisphere. There is no constituency for fiscal responsibility in any of these countries.
Your game here seems to be throwing out simplistic partisan tripe, to either prompt an equally-inflammatory opposite-partisan response that will leave you feeling vindicated, or you can always fall back to arguing the shreds of truth in it while continuing to ignore the big picture.
I would ask you to "do better", but the whole problem is that once you've bought into the contradictions of destructionism, there is no doing better without coming to terms with what you're actually advocating. Talking about an "an Argentina-like reset" you're closer than most, but I suspect that's due to a better ability for rationalization.
In other conversations of ours, you've gone on about the need to preserve American culture in the face of immigration. So once again I will implore you personally, to do your part by working to adopt our culture rather than proudly championing the exact zero-sum politics you're bemoaning (regardless of having changed the color of the partisan flag you're waving). A core tenet of American culture is personal responsibility (for better and for worse), and so these types of choices right here are where the rubber meets the road.
> Your game here seems to be throwing out simplistic partisan tripe
It's not a partisan point at all, it's an observation about how immigration and changes in the polity interacts with politics. Left-wing parties rely on immigrants for votes all over the developed world. And the larger topic of how foreigners influence a democratic system is one Aristotle was writing about even in 350 BC.
> do your part by working to adopt our culture rather than proudly championing the exact zero-sum politics you're bemoaning
You should read up on game theory. It doesn't matter what I do, the millions of other immigrants that come to the U.S. will change its culture to be more like their homelands'. Your only choice is whether you want the left-wing version of those countries' politics, or the right-wing version? Both result in fiscal disaster, followed by an Argentina-style correction. (We should be so lucky--Javier Milei seems to be doing a pretty good job.)
Truthfulness does not imply that something is not partisan. That it's brought up as a talking point to motivate a certain line of thinking is what makes it partisan.
> You should read up on game theory. It doesn't matter what I do
How does this differ from "I might as well drop my trash on the ground in the park, because everyone else is going to do it anyway" ?
You're fallaciously excluding the middle - there will always be some people who drop their trash on the ground. So a clean park must employ groundskeepers to pick it up. But such centralized action can never make up for mass distributed behavior, so it behooves us all to adhere to and propagate good culture that keeps most people from throwing their trash on the ground. And the same dynamic applies to basically every societal construct.
So no, I do not buy your [self-serving] argument that the culture you're simultaneously bemoaning and also engaging in is inevitable.
> To some extend I hope that the Republican party wins the next election. They should be the ones fixing the economy, raising taxes, and doing something responsible for once.
The last small-government Republican was Herbert Hoover. Reagan paid lip service to the concept, but he maintained high spending levels to keep former FDR democrats in the fold.
Core meaning: Barely adequate, average, or just acceptable.
Mediocre means of only ordinary or moderate quality—neither very good nor very bad, and often slightly disappointing
The quality is average but expectations of high quality are not met. He expected more but got what he asked for. We overuse top models because of this.
The claim was there is no difference between Opus and Fable but to me there is.
Opus delivered mediocre results. Not garbage, but would have required me to do lot's of things myself. Fable did not needed my supervision with this task.
> What happens when any AI lab in the world stops caring about this? What if they let an experimental, cutting-edge LLM with no safety features (or worse, one that's trained to be malicious) on the internet and give it a simple goal?
Almost sounds like what those AI safety and alignment people were talking about years ago. The people in these various companies who kept tabs on AI risk out in public and were continuously mocked on HN. All of this stuff is viewed as "future sci-fi" until suddenly it's not.
I've really come to realize recently that there is a very large set of the population of smart people that really has difficulty envisioning future problems unless they directly seem them impacting them today. Otherwise those topics will be continuously dismissed. It explains for me a lot of what I see (both opinions and behaviors) in the broader world that I couldn't understand.
But it is the very people who warned us about rogue AIs going out of control that set up a system that enabled and failed to conrol it.
It is as if Dr Frankenstein continually warned the villagers about monsters then said "Look! See what happened!". No, idiot - YOU sewed the corpses together, YOU set up the lightning collector, and YOU threw the switch.
No, they are two very distinct groups of people who have one commonality, that of talking about rogue AIs. It is as if you are unable to distinguish Dr Waldman from Dr Frankenstein. (https://en.wikipedia.org/wiki/Doctor_Waldman)
Thankyou for the correction, and for continuing the analogy. Unfortunately when the peasants get their pitchforks and torches, they may not distinguish the Dr Waldmans from the Dr Frankenteins either. Hopefully they will.
Anyway, that was not really the point I was trying to get over. These systems that OpenAI and Anthropic and so on are making are not individual AI ('corpses') that have gone out of alignment ('spontaneously revived') and gone wild. They are swarms ('stiched together') and were prompted to do exactly things like this ('struck by lightning'). Ok enough with that analogy, it's dead.
The larger point is that it is unconvincing of these companies to claim that these systems were 'out of control' when they effectively set up a complex system, in the technical sense of a large number of entities with diverse interactions between them. Emergent or surprising behaviour was bound to happen. Then, finally, they prompted it with the equivalent of "hack the world, make no mistakes" then were shocked, shocked that it used all sorts of unexpected tricks to do so.
Ah, I see I was confused - I was thinking of what are now called “AI doomers”, although when I knew them they were called “rationalists”. Everything OpenAI and Anthropic say about rogue AIs, even the terms “alignment”, “AI safety”, “AGI”, they are all cribbed wholesale from what these people were worrying and writing about over the prior two decades. But for most people, they have only heard CEOs of AI companies say this kind of stuff, so that’s who they are thinking of.
I agree completely that the companies are complicit and should have expected exactly this to happen.
"Almost sounds like what those AI safety and alignment people were talking about years ago. The people in these various companies who kept tabs on AI risk out in public and were continuously mocked on HN. All of this stuff is viewed as "future sci-fi" until suddenly it's not."
Are there any practical approaches to AI safety? I hear a lot of warnings but I don't hear much about what to do. Considering that there are many open source models know, what can be done?
Nobody has an answer to alignment and there is no reason to believe that it's the kind of problem you can plausibly solve in one shot against a formidable power-seeking AI.
The closest things to a technical answer I have seen are
1. "We'll have ChatGPT 9 solve it so that ChatGPT 10 is aligned, and then ChatGPT 10 can stop all the other AIs somehow"
2. "Let's do interpretability research so that we can understand what an AI is thinking and then maybe solve the alignment problem with that information."
In terms of non-technical answers, there is
3. hope scaling stops working before we create an AI formidable enough to pose an existential risk
4. hope alignment somehow happens for free
5. hope we can somehow create an enforceable multilateral treaty to stop research into a very profitable enterprise, despite the enormous economic incentives to defect.
I have the most faith in option 3, but unfortunately there's really nothing that can be done to make it more plausible -- it either happens or it doesn't.
"3. hope scaling stops working before we create an AI formidable enough to pose an existential risk"
I have my doubts. The current AI models are already powerful enough to do some real damage. I am always horrified when I read about people giving Claude direct access to a production system and then being wiped out. My use of AI is usually for the AI to propose something which I then review. But that's not very fast so careless people will usually look better. Until something blows up.
And it's only a matter of time until AI even with the current capabilities is being deployed into military or other critical systems.
I think this will go down like any other technology. We'll ignore issues
until there is a real problem. And then hopefully we will do something. Seems with climate change we will soon reach a point where something needs to be done after knowing about consequences already for decades.
We probably also need some massive AI blow ups to (only maybe) do something about it.
Maybe I'm being pessimistic, but we might find ourselves in such a situation that the only practical solution would be to use agents to counter rogue agents. This won't be without collateral damage, though.
Your view sounds more optimistic than mine, honestly. I expect that we will fail to make any serious, coordinated attempt to solve this problem. Then we'll either live or die due to fundamental principles that we currently have no insight into.
Every day I grow more sympathetic to the PauseAI movement, despite the weird hippy vibes. At least they have some ability to rally people together and put boots on the ground in numbers.
> Implement a temporary pause on the training of the most powerful general AI systems, until we know how to build them safely and keep them under democratic control.
Where does this weird idea come that the people actually preaching safety have anything to do with OpenAI or Anthropic? Yes, those companies of course pay lip service to safety, but the Venn diagram of actual AI safety people and big AI corporations is completely disjoint.
OpenAI and Anthropic have published a lot on the need for AI alignment + the research they're doing to ensure alignment/safety, yet they are also responsible for the highest profile misalignment incidents so far (HuggingFace incident, AISI Mythos social engineering, and now this).
One interpretation of this is that they are being deliberately dishonest about their priorities. Another interpretation is that we cannot rely on the labs to self-regulate, because the labs don't trust each other, and there will always be pressure to go to market faster than their competitor.
Either way I think it's pretty non-controversial that the labs are the source of the danger?
> yet they are also responsible for the highest profile misalignment incidents so far
They are the only ones posting about them or admitting to them. That does not mean "the most misalignment incidents so far." You don't know what other attacks have happened (and it's very easy to carry out worse attacks in far higher volume with abliterated GLM 5.3)
Stopping two labs from further research doesn't reduce the danger at all, it just shifts the danger to labs that don't have real safety orgs.
Right, that's why regulation which is universally applied and includes compute controls (to prevent reckless creation of swarms) would be great.
"Posting about or admitting to attacks" is appreciated while people are still unaware of the risks but will be meaningless in the face of an industrial disaster that causes massive amounts of damage or loss of life. At some point, the leading labs must change their development practices, they can't just be allowed to continue rogue agent attacks just because they're willing to admit to them.
looking back at anthropic's promises and committments, the key scaling policies were not upheld. to their credit they have kept the policies up on their website instead of trying to rewrite history. [https://www.anthropic.com/responsible-scaling-policy]
with evidence that committments were not upheld and internal governance has been ineffective, we simply can't trust any such claim made by anthropic or dario amodei.
another reason that neither should be trusted is the lack of remorse or accountability. they are unrepentant. they are not admitting a mistake, they are bragging.
consider the opening line: "Last week, Hugging Face disclosed a new kind of security incident (opens in a new window)." the entire blog is written in the passive voice as if the event was an act of god. you did this.
"We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities". the tone is frankly, excited by it, enthusiastic about it. excited about negligence and criminality.
notice there is no admission of a mistake, no remorse, no apology, nobody held accountable. business as usual. they do not care!
the anthropic blog:
there is not a single admission of a mistake. they are literally, unrepentant.
take the section about the response.
"How we’re responding
We draw several lessons from these incidents.
First, evaluation environments that involve powerful autonomous capabilities also require significant controls."
you learnt that evaluations involving powerful autonomous models require significant controls? you did not realise that autonomous models require significant controls?
it is not a coincidence that this kind of line makes it into the response. the repsonse is laughing at the reader.
the RSP.
simply compare what was promised and what happened. broadly speaking, the rationale of the responsible scaling policy was to stop scaling at certain danger thresholds. in February 2026 they scrapped the policy to stop scaling and now allow themselves to continue scaling regardless of danger. the thing is, they were never going to stop scaling, they were lying. now the part about scaling is gone it is just "the responsible policy".
the claim that the other ai companies also did the same thing is pure speculation.
the law places the burden of proof on the accuser. you can't accuse other companies of crime with no evidence, simply because you don't know if they did it.
Recently? W.r.t. climate this collective denial has been going on for literally decades. With the same patterns. Rationalizing excuses etc. Still going on btw.
Hard to even imagine what that would mean in humans.
reply