AI Weekly Issue #536: Top AI models failed a test of inventing new AI research

* Zero data retention should mean zero; most AI plans suggest a 30-day retention period.
* Epoch AI tested if agents could develop a new machine learning technique independently, requiring 3,000 GPU-hours per agent.
* Agents struggled to improve model handling of images, audio, video, or code repair on the MMPostTrainBench benchmark, resulting in worse models in 52.1% of combinations.
* The median cost for researchers using coding agents reached approximately $601 per day by mid-August, up from under $1 in January.
* Researchers asked five models to reconstruct core ideas of 87 deep learning papers; Claude Fable 5.1 required about 70 yes-or-no hints per idea.
* African leaders requested a larger role in setting global AI standards at a UN Security Council meeting.
* The UK's AI minister noted the lack of energy and data centers to build superintelligence.
* Utah residents participated in an assembly to discuss AI rules, where 40 residents worked out policy options.
* The Pentagon utilizes defense programs like Tradewinds for funding companies pitching AI products via five-minute videos.
* Microsoft is rolling out local models in GitHub Copilot, including an on-device version of MAI Code 1.1 Flash.

The dynamic presented illustrates a tension between accelerating technological capability and the slow, often contested, establishment of governance structures. The observation that AI agents struggle to independently discover new methods, despite massive computational resources, suggests a critical gap between sophisticated pattern recognition and genuine scientific novelty; the performance is often derivative rather than inventive when tested against foundational human knowledge. This technical uncertainty feeds directly into political friction regarding regulation. The split between industry self-policing and calls for external oversight, exemplified by Lina Khan's skepticism toward voluntary pacts versus public testimony from policy leads, reveals a fundamental disagreement on who should bear responsibility: the developers or an external regulatory body. The movement of AI into sensitive domains, such as war planning, highlights how rapidly technical advancements can bypass deliberative, safety-focused processes, risking escalation through systems that amplify human biases and speed up decision loops without requiring superintelligence. Furthermore, the contrast between the technical work being done in isolated labs (agent testing, model reconstruction) and the real-world policy debates taking place in forums like the UN or local assemblies reveals a divergence in focus: one group focuses on emergent capability measurement, while another focuses on establishing enforceable accountability mechanisms for that capability. The pattern suggests that the narrative surrounding AI advancement is being shaped by who controls the data access and what definitions of 'safety' are prioritized—whether safety is defined technically (benchmarks) or politically (governance).
Bridge Questions: If agents demonstrate limited capacity for true novelty, how should regulatory frameworks shift focus from verifying agent outputs to auditing the underlying infrastructure and incentives? What are the specific structural barriers preventing global consensus on AI safety standards when national interests diverge so widely regarding deployment? How can public engagement mechanisms like citizen assemblies be effectively integrated into the high-speed development cycle of frontier AI research?