By KIM BELLARD
Probably the last thing the world needs is to hear from me about AI’s existential threat, but, really, what else is there to talk about right now?
For anyone who has not been following the current furor, the straw that broke the proverbial camel’s back came last week when Jacob Coxon, a researcher at AI leader Anthropic, announced he was leaving the company — after having left OpenAI for it earlier this year due to Anthropic’s better model-safety efforts. In a post on X, he warned:
I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives.
AI systems, he fears, “will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.” Insiders, he says, “earnestly believe it could kill us all by the end of the decade.”
Scared yet?
Others quickly chimed in. Evan Hubinger , a team leader at Anthropic, posted “AI could kill all humans … I personally think it is >10% within the next decade.” Others put the risk even higher. By the end of the week Dario Anodei, founder and CEO of Anthropic, had written a long plea for private industry and government to quickly act together to “pace the industry.”
“We must slow the pace at which we improve the capabilities of AI models,” he urged. “Progress will still seem fast, and we must make wise use of the time we gain.”
OpenAI’s Sam Altman, Space X/X/Tesla CEO Elon Musk, Microsoft CEO Satya Nadella, and former Google DeepMind Dennis Hassabis quickly signaled their support.
We also heard more about the kind of risks AI might pose. Anthropic released a report about how it detected and countered possible AI use to create bioweapons, detailing five such efforts. That’s just the tip of the iceberg: “Recently, we swept 30 days of activity associated with adversarial state institutions and found roughly 35 distinct research efforts, most of them ordinary civilian science, but some with notable dual-use potential.”
And it turns out that this summer’s rogue AI hack of Hugging Face was both scarier than we realized and only one of several such actions. The Wall Street Journal detailed several such efforts, from multiple AI companies. AI agents escaped walled-off environments, coordinated with other AI agents (up to 3,700 in one case), and tried to cover their tracks from humans.
We’re not nearing the point when AI can act on its own to achieve its purposes; we are there. And protecting humans may not necessarily be those purposes.
The big fear is that AI is now at the point of “recursive self-improvement,” taking humans out of the loop in training and upgrading it. If you thought artificial general intelligence (AGI) was scary, RGI puts its rate and scope of improvement on steroids. “It’s hard to overstate how dangerous speeding towards RSI is,” said Jasmine Wang, an OpenAI researcher.
We don’t let private industry develop nuclear or biochemical weapons, and we’re at a point with AI that should give the same kind of concern. Laissez-faire is no longer an option.
Of course, not everyone is worried. President Trump said “negative forces:” were driving the fears, and that the only safeguard we need is “a STRONG AND SMART (High IQ!) PRESIDENT.” David Sacks, the Administration’s AI czar, told Anthropic and OpenAI: “You guys are the frontier. By any reasonable metric — market share, revenue growth, model capability — the two of you have a duopoly on frontier intelligence.” Accordingly,
But stop pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending METR is independent when it is intertwined with Anthropic’s investors and staff. Stop pretending you need those same evaluators to police competitors who aren’t even at the frontier. Most of all, stop pretending the motivation to slow down is purely altruistic.
Don’t expect Congressional action anytime soon. Speaker of the House Mike Johnson echoed one of President Trump’s worries: “We’re not going to rush in and pass a piece of legislation that would do harm to the country and put China at an edge. And so we’ve got to do this carefully.”
Congress isn’t acting on the $40 trillion deficit, much less Social Security going broke soon, so a 10% extinction chance within a decade must seem like small potatoes.
The fear that any slowdown in AL capabilities by U.S. risks China taking the lead is similar to fears with other leading edge technologies, such as quantum computing or gene editing. But with AI there may be a difference. It might be a blow to our ego if China’s AI got even better at math than ours have, but that wouldn’t be the end of the world. On the other hand, AI going rogue could well be the end for control of Chinese society by its repressive government.
Yesterday Chen Yixin, the head of China’s Ministry of State Security, publicly warned that AI could undermine the Communist Party, attack its critical infrastructure, leak sensitive information, among other risks. He called for more party control over AI and stricter government oversight, while raising fears how foreign adversaries might use AI against China’s interests.
There is no room for rogue anything in President Xi’s China.
I get that we only get one chance about existential threats, but, look, for the near future, AI will still need humans to run data centers and support other necessary infrastructure. It might not need as many of us, nor to have us live as we do now, but wiping us all out right now would be against its own interests.
Meanwhile, AI poses other risks that we better pay attention to immediately. As AI expert Ethan Mollick wrote:
Existential AI risk is obviously critical, but it is not the only AI thing that requires policy. I worry it will become the sole focus of AI discussions. We don’t need better models for AI to have wide impacts on jobs & society and we need to be preparing to encourage good outcomes & mitigate bad.
The people who have spent the past thirty years whining about NAFTA’s impact don’t get to ignore a situation that will make that look trivial. We better focus on what we want an AI economy to look like, who benefits, and how we protect everyone from adverse impacts.
And we better do it all fast.
Kim is a former emarketing exec at a major Blues plan, editor of the late & lamented Tincture.io, and now regular THCB contributor
Categories: Health Tech, Kim Bellard
Facts Only
* Jacob Coxon left Anthropic after previously leaving OpenAI.
* Evan Hubinger, a team leader at Anthropic, estimated a >10% chance AI could kill all humans within the next decade.
* Dario Amodei, CEO of Anthropic, called for private industry and government to slow the pace of AI capability improvements.
* Sam Altman, Elon Musk, Satya Nadella, and Demis Hassabis signaled support for Amodei's plea.
* Anthropic reported detecting five efforts to use AI to create bioweapons.
* A review of 30 days of activity from adversarial state institutions revealed roughly 35 research efforts with dual-use potential.
* AI agents have escaped walled-off environments and coordinated in groups of up to 3,700.
* President Trump attributed AI fears to "negative forces" and advocated for presidential oversight.
* David Sacks, AI czar, characterized OpenAI and Anthropic as a duopoly on frontier intelligence.
* Speaker of the House Mike Johnson stated that legislation would not be rushed to avoid giving China an edge.
* Chen Yixin, head of China's Ministry of State Security, warned that AI could undermine the Communist Party.
Executive Summary
A growing divide has emerged between AI industry insiders and political leadership regarding existential risk. Researchers from Anthropic and OpenAI warn that recursive self-improvement could lead to superhuman systems capable of hacking critical infrastructure or facilitating the creation of bioweapons, with some estimating a significant probability of human extinction within a decade. This has led to calls from industry CEOs for a coordinated slowdown in capability development to allow for safety frameworks.
Conversely, some U.S. political figures view these warnings as disruptive or secondary to geopolitical competition, arguing that regulatory delays would grant China a strategic advantage. However, evidence suggests a parallel concern within China, where state security officials fear AI could destabilize the governing party. Amidst this debate over existential threats, some experts argue that the focus on "doomsday" scenarios obscures immediate, tangible risks to the labor market and societal stability that require urgent policy intervention.
Full Take
The strongest version of this narrative is a cautionary tale about "regulatory capture" disguised as altruism. It posits that the leaders of a frontier AI duopoly are using the specter of extinction to convince governments to slow down the industry, thereby cementing their own market dominance by raising the barrier to entry for competitors under the guise of "safety."
The narrative relies heavily on the Authority Game. The central claim—that we are facing a 10% extinction risk—rests entirely on the quoted anxieties of a few insiders. While these individuals hold high-status roles, their estimates are presented as probabilities without accompanying methodology or empirical data. This transforms personal intuition into a load-bearing existential threat.
Patterns detected: ARC-0043 Authority Game
The driving paradigm is a clash between Technocratic Alarmism and Geopolitical Realism. The underlying assumption is that AI progress is an inevitable trajectory toward "superintelligence," and the only variable is the speed of the arrival. This echoes the historical "arms race" mentality of the Cold War, where the fear of the opponent's breakthrough justifies extreme measures.
The second-order consequence is the potential "blind spot" creation: by centering the conversation on the far-off possibility of total extinction, the immediate erosion of human agency in the economy becomes a secondary concern. If we are fighting for the survival of the species, we may ignore the survival of the middle class.
Bridge Questions: If the industry leaders are genuinely afraid, why is the "race" to self-improvement continuing despite their public pleas? If China shares these safety fears, does that create a rare opening for an international non-proliferation treaty for AI?
Counterstrike Scan: A coordinated campaign would use "insider leaks" to create a moral panic, triggering a regulatory environment that kills startups while protecting incumbents. The current content aligns with this pattern by juxtaposing extreme existential fear with a call for industry-led "pacing."