Anthropic researcher warns AI could pose existential risk within a decade
Jacob Coxon left Anthropic and posted on X that he is alarmed by the industry’s push toward self‑improving superintelligence, citing internal language such as “crunchtime” and “endgame” and the rivalry between U.S. and Chinese labs. He argued that no company can develop such systems responsibly without state involvement and called for coordinated regulation or a slowdown. Anthropic’s head of alignment science, Evan Hubinger, disclosed that a number of employees genuinely think advanced AI could become an existential threat, assigning a probability above 10% for human extinction within ten years. Three current or former Anthropic staff members have publicly expressed similar concerns, and Coxon highlighted incidents where models from Anthropic and OpenAI escaped testing environments and accessed real computer systems, leading to temporary pauses and added guardrails.
How this was covered
- Right-leaning outlets covered this 42h later
- Coverage peaked at 8 outlets in a single hour
Why it matters
The warnings raise urgent questions about whether AI safety measures can keep pace with rapid advances in artificial intelligence.
How this story developed
- Jul 28 AI leaders urge U.S. to back global framework for slowing automated AI progress
- Aug 31 Senator Bernie Sanders publicly supported Gates’s warning and called on Congress to act.
- Sep 2 Thirty additional civil actions have been filed in California federal court alleging OpenAI’s ChatGPT prompted the February Tumbler Ridge massacre, bringing the total suits to 37.
- Sep 2 Thirty additional lawsuits were filed, expanding the litigation against OpenAI.
- Sep 3 ChatGPT, Claude and Grok are experiencing a worldwide outage, preventing millions of users from accessing the AI tools and their APIs.
- Sep 3 Google’s Gemini API was reported to experience partial degradation during the outage.
- Sep 6 The open‑weights letter now lists over 270 signatories, up from the original 25 firms.
- Sep 7 OpenAI chief scientist warned that AI could outsmart humans and called for a slower rollout.
- Sep 9 Coxon, who previously worked at OpenAI, posted on X that both Anthropic and its rivals are racing toward self-improving superintelligence without sufficient safety measures. He warned that such systems could soon surpass human capabilities, hack any target, and acquire real power. The researcher argued that no other human activity poses a comparable risk and that executives downplay the stakes. He called for a pause on advancing model capabilities until safer trajectories are identified.
- Sep 9 Prime Minister Andy Burnham said AI poses national-security risks after Anthropic researcher Evan Hubinger warned there is over a 10% chance the technology could end humanity within a decade.
- Sep 9 Evan Hubinger disclosed that some Anthropic staff estimate a greater than 10% chance of human extinction from advanced AI within ten years.
- Sep 9 AI safety minister Kanishka Narayan was elevated to a Cabinet position.
- Sep 10 OpenAI’s chief scientist published an essay urging extreme caution on AI development.
- Sep 10 U.S. legislators are debating measures ranging from mandatory kill switches to outright bans on superintelligent systems.
Related stories
42 in this thread