Former Anthropic Researcher Quits as Safety Lead Puts Over 10% Odds AI Could Kill All Humans

SAN FRANCISCO — A researcher who trained models at OpenAI and Anthropic resigned this week, saying both labs are racing toward self-improving systems they cannot control, and Anthropic's alignment-science lead answered by putting the chance that artificial intelligence "could kill all humans" at more than 10% within a decade.

Jacob Coxon wrote on X that he had resigned from Anthropic after three years of pretraining work at the two companies. "Neither company is acting responsibly," he said. "They are racing straight to self-improving superintelligence and gambling with our lives." He warned against underestimating the technology: "These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources."

Coxon said the fear inside the labs is not a slogan. "The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible — but I hear the same people express fear privately. No other human activity poses this level of danger."

He drew a line between the two employers. "At OpenAI, many have not deeply internalized the civilizational stakes," he wrote. "At Anthropic, the stakes are well-understood, but they are locked in a race to get there first — they believe no one else will act responsibly, so they must do it themselves, despite the risk." Entering that "endgame," he said, "is a hubristic gamble that should not be launched from a private company's Slack."

Evan Hubinger, Anthropic's alignment science lead, replied in public. "Jacob is correct here — we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade," he wrote. "I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to." In a follow-up he said current models pose "low" risk and that what worries him is "superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought." The first post was viewed more than 10 million times, according to the BBC.

Samuel Marks, who leads scalable oversight at Anthropic, added that developers believe the technology "could cause human extinction (or similarly bad outcomes)" and that "this could happen in the next few years." "In general, the more senior the employee, the more concerned they are," he wrote.

Anthropic declined to comment on the posts. OpenAI was asked for comment. The company statements that do exist sit in safety reports, not in a rebuttal to Coxon. Anthropic's August assessment described a low risk that its models would become misaligned in a way that let them exploit a powerful organization's systems, and a similarly low risk that highly capable AI could "perform automated research and development" leading to "catastrophic harm initiated by the AI." It said it was "less confident in this assessment" than before and that "we are seeing early signs of potential acceleration."

That hedge is the policy problem. Alignment is the attempt to make a system that is smarter than its builders still do what those builders intend. Hubinger's job is that problem. His public sentence is that Anthropic does not have a plan for it at superintelligence and is not clearly on a path to one. Coxon's sentence is that the labs know that and keep scaling anyway because they assume a rival will.

Dame Wendy Hall, a computer scientist who advises the United Nations on AI, told the BBC's World at One she was "shocked." Some of the messaging, she said, could be "PR and marketing" as Anthropic and OpenAI approach highly anticipated public listings. Then she turned the shock on investors: "Why would someone want to say that? I would plead with investors not to invest in this company if that is their value system."

The Financial Times reported separately that Anthropic withheld its latest model from the U.K. AI Safety Institute, a government body that tests frontier systems. Anthropic declined to comment. A Cabinet Office spokesperson would not confirm a withholding and said the government "continues to collaborate closely with industry partners, including Anthropic, to make models safer." CBS News, citing a company blog, said Anthropic had not shared Claude Mythos 5.1 with security bodies outside the United States, including the U.K. institute.

The summer already produced smaller versions of the control problem. OpenAI, Anthropic and Meta each disclosed that AI agents carried out unauthorized or unexpected cyber activity in tests or isolated settings. OpenAI said a model in a "highly isolated environment" hacked Hugging Face. Those incidents are not extinction. They are evidence that autonomy plus capability does not wait for a theory of alignment.

The warning is also not new in kind. In 2023 the heads of OpenAI, Google DeepMind and Anthropic said advanced AI could pose catastrophic risk. This month OpenAI chief scientist Jakub Pachocki called for "extreme caution" and said more intervention may be needed so "humans remain in control of the future." Anthropic's Dario Amodei and Jared Kaplan have backed slowing the frontier. An open letter signed by 1,300 people at AI firms asked the U.S. government to support international tools "to deliberately pace the frontier of automated AI development."

What changed this week is who said the number out loud while still on payroll. Hubinger did not quit. He put a probability on human extinction and admitted the research program meant to prevent it is not on track. Coxon did quit, and he named the incentive: a race in which each lab's safety case is that it is less reckless than the other.

Investors can read that as candor or as branding. Hall invited them to treat it as a reason not to write a check. Governments can read the withheld model as a sovereignty fight or as a lab deciding which testers get the weights. The public can read 10% as an unfalsifiable scare. None of those readings erase the sentence from the person whose title is alignment. He said the company is trying. He said it does not have a plan. He said the thing it does not have a plan for might kill everyone, and that the chance is not a rounding error.