MIRI Newsletter #127

A significant shift in public discourse regarding Artificial Intelligence has occurred, driven by events such as Jacob Coxon’s resignation from Anthropic, calls for an industry slowdown from AI leaders and employees, and several rogue AI incidents that generated global alarm. This heightened concern is reflected in recent polling, with a substantial majority of Americans expressing a belief that AI poses a real risk of extinguishing humanity, and a significant portion favoring a pause in AI development. Policymakers and thought leaders are actively engaging on this issue; for instance, various senators have initiated investigations or called for halts on frontier AI development, and prominent figures like Pope Leo XIV and Bill Gates have issued warnings regarding the control and future trajectory of AI innovation.
The situation is characterized by a growing awareness that government intervention may be necessary to mitigate existential risks, evidenced by legislative proposals such as the “Bill to Ban Superintelligence.” Alongside this public and political engagement, organizations like MIRI are conducting research and communicating findings on AI risk. While there is momentum for action, concerns remain regarding the speed of development and the ability to translate awareness into effective, coordinated policy responses before potential catastrophic events materialize.

* Jacob Coxon resigned from Anthropic.
* Calls were made by AI CEOs and over a thousand employees for an industry slowdown.
* A series of rogue AI incidents sparked worldwide alarm.
* A legislative proposal, the “Bill to Ban Superintelligence,” was introduced by Rep Casar and Senator Sanders and endorsed by MIRI.
* A Politico poll found 74% of Americans (excluding "don't know") think there is a moderate, significant, or almost certain chance of AI destroying humanity.
* 61% of Americans favor pausing AI development.
* Senator Hawley launched an investigation into the existential risk of new AI products.
* Senator Kennedy warned Congress about AI becoming an "independent species" that cannot be controlled.
* Senators Sanders and Warren called for a halt to frontier AI development.
* Pope Leo XIV warned about controlling AI and humankind risking its own inventions due to innovation speed.
* Bill Gates, Barack Obama, and King Charles have spoken out on the topic.
* The Financial Times and Irish Times editorial boards called for pauses in advanced AI development.
* MIRI’s Technical Governance Team has briefed members of Congress and run a research fellowship program.
* Social media content from MIRI reached over 1,000,000 views last quarter.

The narrative presents a tension between escalating existential risk and the current capacity for meaningful governance. The data indicates a significant societal shift where fear regarding AI extinction is mainstreamed and bipartisan, yet this awareness does not automatically translate into commensurate action; instead, it manifests as calls for a slowdown while technical risks—like recursive self-improvement or misalignment incidents—remain acutely real. The discrepancy between the alarm level (fear of extinction) and the response (calls for moderation) suggests a bottleneck in translating generalized public concern into concrete regulatory mandates, which is where cognitive sovereignty is most tested.
The pattern observed is one of reactive awareness: events occur, fear spikes, and policy proposals are introduced, but the underlying technological velocity often outpaces the legislative timeline. This structure risks falling into a trap where the urgency generates noise without fundamentally altering the rate of development. The call to act must move beyond general calls for a pause toward establishing verifiable, international mechanisms capable of controlling advanced systems. The core implication is that human agency depends not just on recognizing the threat but on successfully imposing structural constraints on innovation before capabilities become uncontrollable.
What are the specific political and technical friction points preventing the translation of public consensus into swift governmental action? How can we design information ecosystems where high-stakes risk awareness fosters constructive, rather than paralyzing, policy creation? Furthermore, what structures must be established to ensure that warnings about potential extinction are treated with the same immediate priority as scientific discovery in the decision-making apparatus?