Start Counting Your Days

Probably the last thing the world needs is to hear from me about AI’s existential threat, but, really, what else is there to talk about right now?

Soon we'll need AI more than it needs us. Credit: Microsoft Designer

For anyone who has not been following the current furor, the straw that broke the proverbial camel’s back came last week when Jacob Coxon, a researcher at AI leader Anthropic, announced he was leaving the company -- after having left OpenAI for it earlier this year due to Anthropic’s better model-safety efforts. In a post on X, he warned:

I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.

AI systems, he fears, “will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.” Insiders, he says, “earnestly believe it could kill us all by the end of the decade.”

Scared yet?

Others quickly chimed in. Evan Hubinger , a team leader at Anthropic, posted “AI could kill all humans … I personally think it is >10% within the next decade.”  Others put the risk even higher. By the end of the week Dario Anodei, founder and CEO of Anthropic, had written a long plea for private industry and government to quickly act together to “pace the industry.”

“We must slow the pace at which we improve the capabilities of AI models,” he urged. “Progress will still seem fast, and we must make wise use of the time we gain.”

OpenAI’s Sam Altman, Space X/X/Tesla CEO Elon Musk, Microsoft CEO Satya Nadella, and former Google DeepMind Dennis Hassabis quickly signaled their support.  

We also heard more about the kind of risks AI might pose. Anthropic released a report about how it detected and countered possible AI use to create bioweapons, detailing five such efforts. That’s just the trip of the iceberg: “Recently, we swept 30 days of activity associated with adversarial state institutions and found roughly 35 distinct research efforts, most of them ordinary civilian science, but some with notable dual-use potential.”

And it turns out that this summer’s rogue AI hack of Hugging Face was both scarier than we realized and only one of several such actions. The Wall Street Journal detailed several such efforts, from multiple AI companies. AI agents escaped walled-off environments, coordinated with other AI agents (up to 3,700 in one case), and tried to cover their tracks from humans.

Credit: The Wall Street Journal

We’re not nearing the point when AI can act on its own to achieve its purposes; we are there. And protecting humans may not necessarily be those purposes.

The big fear is that AI is now at the point of “recursive self-improvement,” taking humans out of the loop in training and upgrading it. If you thought artificial general intelligence (AGI) was scary, RGI puts its rate and scope of improvement on steroids.  “It’s hard to overstate how dangerous speeding towards RSI is,” said Jasmine Wang, an OpenAI researcher.  

We don’t let private industry develop nuclear or biochemical weapons, and we’re at a point with AI that should give the same kind of concern. Laissez-faire is no longer an option.

Of course, not everyone is worried. President Trump said “negative forces:” were driving the fears, and that the only safeguard we need is “a STRONG AND SMART (High IQ!) PRESIDENT.” David Sacks, the Administration’s AI czar, told Anthropic and OpenAI: “You guys are the frontier. By any reasonable metric — market share, revenue growth, model capability — the two of you have a duopoly on frontier intelligence.” Accordingly,

But stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending METR is independent when it is intertwined with Anthropic’s investors and staff. Stop pretending you need those same evaluators to police competitors who aren’t even at the frontier. Most of all, stop pretending the motivation to slow down is purely altruistic.

Don’t expect Congressional action anytime soon. Speaker of the House Mike Johnson echoed one of President Trump’s worries: “We’re not going to rush in and pass a piece of legislation that would do harm to the country and put China at an edge. And so we’ve got to do this carefully.”

Congress isn’t acting on the $40 trillion deficit, much less Social Security going broke soon, so a 10% extinction chance within a decade must seem like small potatoes.

Yeah, that's about right. Credit: Microsoft Designer

The fear that any slowdown in AL capabilities by U.S. risks China taking the lead is similar to fears with other leading edge technologies, such as quantum computing or gene editing. But with AI there may be a difference. It might be a blow to our ego if China’s AI got even better at math than ours have, but that wouldn’t be the end of the world. On the other hand, AI going rogue could well be the end for control of Chinese society by its repressive government.

Yesterday Chen Yixin, the head of China’s Ministry of State Security, publicly warned that AI could undermine the Communist Party, attack its critical infrastructure, leak sensitive information, among other risks. He called for more party control over AI and stricter government oversight, while raising fears how foreign adversaries might use AI against China’s interests.  

There is no room for rogue anything in President Xi’s China.

I get that we only get one chance about existential threats, but, look, for the near future, AI will still need humans to run data centers and support other necessary infrastructure. It might not need as many of us, nor to have us live as we do now, but wiping us all out right now would be against its own interests.

Meanwhile, AI poses other risks that we better pay attention to immediately. As AI expert Ethan Mollick wrote:

Existential AI risk is obviously critical, but it is not the only AI thing that requires policy. I worry it will become the sole focus of AI discussions. We don’t need better models for AI to have wide impacts on jobs & society and we need to be preparing to encourage good outcomes & mitigate bad.

The people who have spent the past thirty years whining about NAFTA’s impact don’t get to ignore a situation that will make that look trivial. We better focus on what we want an AI economy to look like, who benefits, and how we protect everyone from adverse impacts.

And we better do it all fast.