Former Anthropic researcher quits over fears superintelligence could wipe out humanity

Former Anthropic researcher quits over fears superintelligence could wipe out humanity

N
News Editor
2026-09-10 05:45:00
Jacob Coxon, a 27-year-old AI researcher who worked on large-model pretraining at both OpenAI and Anthropic, said he resigned because he believes leading labs are racing toward self-improving superintelligence while effectively betting human lives on the outcome. His post on X drew more than 100 million views, according to the source article, and reignited debate over existential AI risk inside top model companies. The PANews report says Coxon was listed as a contributor to GPT-4o and a core contributor to GPT-4.5 before moving to Anthropic this year. After his post, Anthropic’s alignment lead publicly backed him and said the probability of such a failure scenario happening within 10 years was greater than 10%. Another Anthropic safety researcher also supported him in a personal capacity, while former scalable oversight lead Joe Benton said Coxon’s description of the industry was accurate. The article places Coxon’s warning alongside Tim Urban’s 2015 essay “The AI Revolution,” revisiting ideas such as exponential progress, intelligence gaps, recursive self-improvement, and the possibility that advanced AI could push humanity toward either extinction or what Nick Bostrom called “species immortality.”

Jacob Coxon, a 27-year-old AI researcher who recently left Anthropic, said on X that he resigned because he believes OpenAI and Anthropic are rushing toward self-improving superintelligence and, in his words as cited by the source article, are gambling with everyone’s lives.

Former Anthropic researcher quits over fears superintelligence could wipe out humanity 2

The post surpassed 100 million views on X, according to the PANews article. Coxon said he had spent the past three years working on large-model pretraining at OpenAI and Anthropic, and wrote that a number of people directly involved in building frontier AI privately believe that if things go badly, humanity could be wiped out before the end of this decade.

Coxon’s background and why his comments drew attention

The article says Coxon is British, 27 years old, and trained in mathematics. He made the UK team for the International Mathematical Olympiad in 2016 and 2017, then studied mathematics at Trinity College, Cambridge.

In 2021, he took part in a medical statistics paper published in eLife that used Bayesian methods to study the relationship between genotype and survival rate in patients with tuberculous meningitis.

He joined OpenAI around 2023, the article says. His name appeared on the official contributor list for GPT-4o, and OpenAI later listed him as a core contributor to GPT-4.5. This year he moved to Anthropic, where he continued working on core pretraining efforts.

PANews says he did not stay long at Anthropic. Unable to keep doing the work against his own judgment, he quit and posted his warning on X. The article also says The Wall Street Journal reported that he may leave the AI industry entirely.

Former Anthropic researcher quits over fears superintelligence could wipe out humanity 3

Support from Anthropic researchers

After Coxon’s post, Anthropic’s alignment lead publicly backed him. According to the article, that executive said Coxon was right and put the probability of such an outcome within 10 years at more than 10%.

Another Anthropic researcher working on safety also voiced support in a personal capacity. Based on that person’s own observation, the article says, the more senior someone is inside AI companies, the more likely they are to worry about this problem.

Joe Benton, who recently left Anthropic and had previously led work on scalable oversight, also said Coxon’s description of the industry’s current state was accurate.

The report adds an important qualifier: these are still judgments about future risk. No one knows whether the 10% figure is right.

Why the article links this moment to a 2015 essay

The PANews piece connects the latest dispute to Tim Urban’s January 2015 essay, “The AI Revolution,” which was split into two parts: “The Road to Superintelligence” and “Our Immortality or Extinction.”

Former Anthropic researcher quits over fears superintelligence could wipe out humanity 4

As the article frames it, the essay offered a blunt proposition: if humanity eventually builds superintelligence that far exceeds human capability, it may push civilization in only two directions. One is extinction. The other is immortality.

After seeing Coxon’s post, the author says he revisited Urban’s essay and came away with the sense that today’s developments look like an extension of the timeline laid out there.

One of Urban’s core ideas, as recounted in the article, was that humans struggle to understand exponential growth because the mind defaults to linear projection. Civilization, though, does not always move that way. At certain points it accelerates sharply.

To illustrate that point, the piece repeats one of Urban’s examples. If someone from 1750 were transported to 2015 and shown airplanes, highways, smartphones, the internet, satellites, nuclear weapons, and space stations, that person might feel as if they had entered a different world. But if that same person from 1750 went back another 265 years and brought someone from 1485 forward, the shock would be much smaller. Going further back, it might take tens of thousands of years to create a similar gap in technological progress.

Urban used the term DPU, or Die Progress Unit, for a chunk of progress large enough to completely overwhelm an ancient observer. The article says one DPU might once have taken 10,000 years to accumulate, then a few hundred years, then decades, with the interval shrinking over time.

Former Anthropic researcher quits over fears superintelligence could wipe out humanity 5

Examples the article uses to argue AI progress now feels different

The piece reviews a run of milestones. In 2022, ChatGPT showed that AI could talk in a human-like way. In 2023, GPT-4 broadened public awareness of multimodal large models. After that came reasoning models represented by o1, then Claude Code, Codex, and agents in quick succession.

The article also says that as of last week, GPT-6 Astra could operate a computer like a person, opening Blender, Houdini, Unity, and Excel, finding buttons in unfamiliar software, trying different actions, and continuing for hours.

On the research side, the article lists several cases. In May, an OpenAI model overturned an Erdős discrete geometry conjecture that had stood for 80 years. On Sept. 4, Anthropic used Claude over 11 days to convert Fermat’s Last Theorem into a 13 million-line formal proof that can be checked line by line by a computer, a task that the article says humans might otherwise have needed 10 years to complete. It also mentions the recent controversy over OpenAI reportedly solving a Millennium Prize problem in 88 hours.

These examples are not presented as Coxon’s only concern. The PANews article says the deeper issue is self-improving superintelligence.

Self-improvement, intelligence gaps, and positive feedback loops

Urban asked in 2015 what would happen if AI began helping humans build stronger AI. The article says that by February 2026, OpenAI described GPT-5.3 Codex as its first model to play a key role in its own creation.

Former Anthropic researcher quits over fears superintelligence could wipe out humanity 6

It also says that nearly every stronger model now has AI involved in the training process. For the author, that makes an idea that once looked speculative feel much closer to the present.

Another part of Urban’s framework, as summarized here, is the “intelligence ladder.” People tend to underestimate superintelligence because they imagine a narrow gradient: ordinary person, smart person, genius, Einstein, super-AI. The article argues that a better comparison is the gap between chimpanzees and humans. On the scale of biological intelligence, the difference may not look enormous, yet one side uses sticks to probe termite nests while the other invents calculus, quantum mechanics, and the internet, and leaves Earth.

From there the article returns to “intelligence explosion.” If AI reaches the level of top AI researchers, one of its most valuable jobs would naturally be AI research itself, helping humans build the next generation of systems. Once that next generation becomes better at designing another generation, intelligence gains and the ability to create greater intelligence start reinforcing each other.

The piece is explicit that the world is still far from full recursive self-improvement. “GPT-6 won’t build GPT-7 by itself, and GPT-7 won’t wake up in the middle of the night and train GPT-8,” the article says in essence. But it argues that the earliest links in that chain are already visible.

Capital, GPUs, and competitive pressure

The article describes the dynamic as a powerful inertia. Smarter AI creates greater economic value. Greater economic value attracts more capital. More capital buys more GPUs. More GPUs train stronger AI. Stronger AI then helps humans build even stronger AI.

Former Anthropic researcher quits over fears superintelligence could wipe out humanity 7

That loop unfolds under competition between companies and between countries. As the article puts it, if one group stops, others keep going. If one group does not move forward, others will. In that framing, civilization looks like a river that keeps getting steeper, while no single person or institution can fully control where it flows.

Extinction or species immortality

The essay then returns to a concept attributed to Nick Bostrom. Earth has seen countless species, and almost all of them disappeared. Extinction, in that view, acts like an attractor state. Once a species falls into it, there is no return. Superintelligence, however, might open a second attractor: species immortality.

Urban illustrated the idea with a balance beam, the article says. Humanity has been walking on it all along. Keep walking long enough and something eventually knocks us off: an asteroid, a pandemic, war, or a technology of our own making. If superintelligence arrives, that could be the hardest collision yet. On one side is the ending most species already know, extinction. On the other is a destination life on Earth has never reached, immortality.

Why Coxon’s resignation resonated

Read on a civilizational timescale, the article argues, AI is no longer only a question of whether programmers lose jobs, what happens to designers, or which company wins. It becomes a species-level wager: can AI help humanity escape death as an ultimate rule, or does the bet end in extinction, perhaps accelerated extinction?

That is the frame in which the PANews author says Coxon’s resignation makes sense. At the same time, the article says it also understands why many people who are worried still stay in labs and keep working. The other side of the wager is extraordinarily attractive.

Former Anthropic researcher quits over fears superintelligence could wipe out humanity 8

AI is already taking part in mathematical discovery, drug development, human genome analysis, new materials research, and the operation of experiments and software, according to the article. If that curve keeps advancing for another three or five years, the author says, no one can say with confidence what follows.

The article’s closing view

The piece ends by arguing that AI is gradually moving beyond the category of an ordinary technology product. It looks more like the first time a species has tried to create something smarter than itself.

The author does not claim certainty. Instead, he says he hopes the balance beam Urban described ultimately tips humanity to the other side: a world where cancer truly disappears, aging becomes a treatable condition, people live much longer lives, humanity leaves the Solar System for Alpha Centauri, and warnings that AI would destroy humanity are remembered as fears that did not materialize.

If AI really helps humanity cross a threshold that no species has crossed in millions of years, freeing life from disease, aging, random catastrophe, and death, then, the article concludes, it would become the greatest event in human history.

This article was originally published by Bit.Fan. For more cryptocurrency news and market insights, visit www.bit.fan.
1400

Disclaimer:

The market information, project data, and third-party content displayed on this platform are for industry information sharing only and do not constitute any form of investment advice or return commitment.

Cryptocurrency trading carries high risks. Users should fully assess their risk tolerance and make independent decisions. All profits, losses, and legal responsibilities are borne by the users themselves.