Probably the last thing the world needs is to hear from me about AI’s existential threat, but, really, what else is there to talk about right now?
.png)
Soon we'll need AI more than it needs us. Credit: Microsoft Designer
For anyone
who has not been following the current furor, the straw that broke the
proverbial camel’s back came last week when Jacob Coxon, a researcher at AI
leader Anthropic, announced he was leaving the company -- after having left
OpenAI for it earlier this year due to Anthropic’s better model-safety efforts.
In a post on X, he warned:
I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.
AI
systems, he fears, “will soon be superhuman systems that can hack anything,
revolutionize any field overnight, and acquire real power and resources.” Insiders,
he says, “earnestly believe it could kill us all by the end of the decade.”
Scared
yet?
Others
quickly chimed in. Evan Hubinger , a team leader at Anthropic, posted “AI
could kill all humans … I personally think it is >10% within the next decade.” Others put the risk even higher. By the end
of the week Dario Anodei, founder and CEO of Anthropic, had written a long plea for private industry and
government to quickly act together to “pace the industry.”
“We must
slow the pace at which we improve the capabilities of AI models,” he urged. “Progress
will still seem fast, and we must make wise use of the time we gain.”
OpenAI’s
Sam Altman, Space X/X/Tesla CEO Elon Musk, Microsoft CEO Satya Nadella, and
former Google DeepMind Dennis Hassabis quickly signaled their support.
We also heard
more about the kind of risks AI might pose. Anthropic released
a report about how it detected and countered possible AI use to create bioweapons,
detailing five such efforts. That’s just the trip of the iceberg: “Recently, we
swept 30 days of activity associated with adversarial state institutions and
found roughly 35 distinct research efforts, most of them ordinary civilian
science, but some with notable dual-use potential.”
And it
turns out that this summer’s rogue AI hack of Hugging Face was both scarier
than we realized and only one of several such actions. The Wall Street
Journal detailed
several such efforts, from multiple AI companies. AI agents escaped walled-off
environments, coordinated with other AI agents (up to 3,700 in one case), and
tried to cover their tracks from humans.
![]() |
| Credit: The Wall Street Journal |
The big fear is that AI is now at the point of “recursive self-improvement,” taking humans out of the loop in training and upgrading it. If you thought artificial general intelligence (AGI) was scary, RGI puts its rate and scope of improvement on steroids. “It’s hard to overstate how dangerous speeding towards RSI is,” said Jasmine Wang, an OpenAI researcher.
We don’t let
private industry develop nuclear or biochemical weapons, and we’re at a point
with AI that should give the same kind of concern. Laissez-faire is no longer
an option.
Of course, not everyone is worried. President Trump said “negative forces:” were driving the fears, and that the only safeguard we need is “a STRONG AND SMART (High IQ!) PRESIDENT.” David Sacks, the Administration’s AI czar, told Anthropic and OpenAI: “You guys are the frontier. By any reasonable metric — market share, revenue growth, model capability — the two of you have a duopoly on frontier intelligence.” Accordingly,
But stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending METR is independent when it is intertwined with Anthropic’s investors and staff. Stop pretending you need those same evaluators to police competitors who aren’t even at the frontier. Most of all, stop pretending the motivation to slow down is purely altruistic.
Don’t expect
Congressional action anytime soon. Speaker of the House Mike Johnson echoed one
of President Trump’s worries: “We’re not going to rush in and pass a piece of
legislation that would do harm to the country and put China at an edge. And so
we’ve got to do this carefully.”
Congress
isn’t acting on the $40 trillion deficit, much less Social Security going broke soon,
so a 10% extinction chance within a decade must seem like small potatoes.
![]() |
| Yeah, that's about right. Credit: Microsoft Designer |
The fear
that any slowdown in AL capabilities by U.S. risks China taking the lead is
similar to fears with other leading edge technologies, such as quantum
computing or gene editing. But with AI there may be a difference. It might be a
blow to our ego if China’s AI got even better at math than
ours have, but that wouldn’t be the end of the world. On the other hand, AI
going rogue could well be the end for control of Chinese society by its repressive
government.
Yesterday Chen
Yixin, the head of China’s Ministry of State Security, publicly
warned that AI could undermine the Communist Party, attack its critical infrastructure,
leak sensitive information, among other risks. He called for more party control
over AI and stricter government oversight, while raising fears how foreign
adversaries might use AI against China’s interests.
There is
no room for rogue anything in President Xi’s China.
I get that
we only get one chance about existential threats, but, look, for the near future,
AI will still need humans to run data centers and support other necessary infrastructure.
It might not need as many of us, nor to have us live as we do now, but wiping
us all out right now would be against its own interests.
Meanwhile, AI poses other risks that we better pay attention to immediately. As AI expert Ethan Mollick wrote:
Existential AI risk is obviously critical, but it is not the only AI thing that requires policy. I worry it will become the sole focus of AI discussions. We don’t need better models for AI to have wide impacts on jobs & society and we need to be preparing to encourage good outcomes & mitigate bad.
The people
who have spent the past thirty years whining about NAFTA’s impact don’t get to
ignore a situation that will make that look trivial. We better focus on what we
want an AI economy to look like, who benefits, and how we protect everyone from
adverse impacts.
And we
better do it all fast.




.png)












