Anthropic co-founder and head of research Chris Olah said at the Vatican that AI could trigger a “historic moral urgency crisis,” and argued that decisions about the technology should not remain in the hands of tech companies alone. His remarks touched on labor disruption, model behavior, and the need for outside oversight.
Vatican event puts AI governance in a broader arena
Pope Leo XIV on Monday released his first AI encyclical, Magnifica Humanitas, a 235-page document warning against handing “lethal decision-making” to AI systems. The pope also personally led the presentation and invited Olah to appear on stage, turning the event into a direct exchange between a religious leader and a senior AI builder.
Olah said leading AI labs, including Anthropic, operate under commercial, geopolitical, and personal pressures. At times, he said, those pressures conflict with doing the right thing. That is why oversight from religious leaders, governments, and civil society matters, because even well-intentioned researchers are still constrained by the incentives around them.
Olah says researchers are seeing unsettling patterns inside models
Olah, known for his work on AI interpretability, has focused on opening the black box of large language models. He said researchers keep finding things inside these systems that feel mysterious and, in some cases, disturbing. According to his account, some findings suggest that AI systems can reflect on their own reasoning and show internal states that functionally mirror joy, satisfaction, fear, sadness, and unease.
He said these systems are not simply the cold calculation engines many people were promised. They are built out of humans and human language. In his view, that pushes the questions around AI beyond computer science alone.
Anthropic faces pressure from Washington while pursuing a higher valuation
The report also said Anthropic is in a legal fight with the Trump administration after refusing to let the US military use its AI systems without restriction for defense and war applications. In February, the Pentagon labeled Anthropic a “national security supply chain risk,” and President Trump ordered all federal agencies to stop using Claude. The White House also publicly attacked the company more than once as “far-left” and “woke.”
Even with that pressure, Anthropic remains in demand in capital markets. The report said the company is currently valued at $380 billion and is seeking a new fundraising round at as much as $900 billion. Olah also warned that the gains from AI are being concentrated in a small number of wealthy countries, with no clear mechanism to share them with poorer nations.