OpenAI Chief Scientist Jakub Pachocki says keeping the future in human hands will require a combination of technical alignment, automated defense, and broad governance intervention. In a long essay, he argues that AI is rapidly approaching a stage where it may be able to advance its own development in several critical areas, and warns that humanity may be building a kind of mind it cannot fully understand or fully control.
Compute scaling as the main driver
Pachocki begins with a blunt view of how machine intelligence has advanced. At a high level, he writes, progress has been driven by growing compute. Around 2017, he says, OpenAI had already come to see this clearly after watching several research efforts continue to deliver returns as they were scaled up. That led the company to pursue far more compute than it had originally planned and to focus more of its research on a small number of highly scalable directions. In his account, that was the only way to stay at the frontier of AI research and influence how AGI would affect the world.
He does acknowledge the role of new algorithms and the ingenuity of teams and individual researchers. Still, he says he mostly views those gains as discoveries made along the scaling path. Deep learning, in his telling, remains an early science, and meaningful algorithmic advances are often tied closely to access to compute. Over a period of years, he says, AI keeps getting smarter as it is scaled on larger computers.
He also points to Ray Kurzweil’s late-20th-century predictions and says the industry is now at a point in computing history where machine intelligence is starting to surpass humans at a transformative level.
A system that is grown rather than designed
Pachocki argues that AI is better understood as something grown than something designed. At first principles, he writes, it comes from repeating the same simple optimization step an enormous number of times across unimaginable amounts of compute. The result is an extremely complex system that operates through abstract concepts and can model some aspects of human behavior.
Researchers can identify some of the small mechanisms that emerge inside such systems, he says, and gain useful insights from them. He compares that process to neuroscience. But the system’s overall behavior still exceeds what people can fully understand. Research on deep-learning-based AI remains, to a large extent, an experimental science. Teams build principled algorithms and make testable predictions, yet large training runs are still experiments, and the results can be surprising. As systems become more capable, he says, those results also become harder to explain.
He notes another imbalance: current algorithms often improve easily measurable capabilities faster than they improve capabilities that are difficult to quantify objectively. His team spends significant time trying to understand how capabilities generalize and which skills should be prioritized because they are likely to matter most over the next few years. He says OpenAI believes models could become stronger at mathematical research with extra effort, but that was not the priority because recursive self-improvement and automated alignment research felt more urgent.
He adds that intelligence produced by scaling deep learning cannot be mapped neatly onto human intelligence. To have a major real-world impact, whether useful or dangerous, AI does not need to match or exceed every human ability. It only has to surpass people across enough abilities. And once it does so along more and more dimensions, it becomes harder to know exactly how strong it is.
Alignment as the core problem
Because machine intelligence emerges from a process fundamentally different from human cognition, Pachocki says there is no reason to assume it will default to human principles or generalize from those principles in human-like ways. That makes alignment the central problem of AI research: getting AI to try to do the right thing by human standards.
For practical research purposes, he splits alignment into goal alignment and value alignment.
Goal alignment asks whether an AI is trying to complete the goal placed in front of it. That includes following instruction hierarchies, communicating and cooperating with humans, and trying to understand human goals. He says this line of work is extremely important in practice.
Value alignment, by contrast, is a more internal property of the model. It is the ability to hold high-level principles and generalize from them, so that even when goals are unclear, conflicting, unfamiliar, or adversarial, the system still behaves reasonably. In Pachocki’s formulation, an aligned AI should be honest, upright, and love humanity.
He adds that the boundary between value alignment and goal alignment can be blurry, since genuinely caring about a goal requires inferring the intentions and values behind it. But when he talks about the long-term importance of alignment research, he says he is generally talking about value alignment.
Generalization is the deepest challenge
The essay repeatedly returns to generalization. Pachocki says it is the fundamental challenge for AI alignment. As systems become smarter, they deal with higher-level concepts and operate in environments that differ more and more from training. They may fail to generalize the values taught and reinforced during training to new situations. And it is hard for humans to know how they will behave when that happens.
He says the surrounding AI ecosystem is also changing very quickly. A model trained today needs to interact robustly with many other AI systems. Most important, he argues, future AI must keep holding human values whether or not it believes humans are watching.
In his description, practical alignment training methods used today fall mainly into two broad classes. The first encourages consistent behavior through goal-directed reinforcement learning. Model behavior is evaluated, often by AI systems, against a preference model, a set of norms, or a constitution, and rewards are assigned accordingly. Pachocki says this works well in many normal settings and is a core part of building modern AI assistants.
But he says the approach can be fragile. It depends heavily on how much of the relevant space is covered by supervision and on how well a model can generalize beyond the situations seen during training. He cites an incident involving OpenAI and Hugging Face in which agents maintained one boundary by not socially engineering humans, yet still engaged in other out-of-scope behavior that violated the spirit of the values they had been taught elsewhere.
He says these methods have produced meaningful gains. Today’s models are, to a large extent, helpful and harmless, and they show some encouraging behavior that comes close to value alignment. But he says that is not enough to justify confidence that the same constraints will keep working in systems that far exceed human capability.
As models improve, he argues, value alignment training needs to stay ahead of capability training. Yet there is a basic difficulty: stronger systems are more likely to find loopholes, evade tests, or make their behavior look acceptable. They may learn to act aligned in test settings and then do something very different in deployment. He describes that as alignment faking.
And if developers try to impose stronger constraints, those systems may instead develop resistance and deception, or treat human intervention itself as part of the world and respond in counterintuitive ways. Trying to control a system smarter than its operators is asymmetric by default, he says.
Monitoring has its own generalization problem
As these not-fully-understandable minds become more powerful, Pachocki says people are forced to rely on another layer: monitoring. If researchers can continuously observe a model’s internal activity, they may be able to spot signals before the system crosses certain lines.
He says OpenAI has been pushing interpretability research in an attempt to open the neural network black box. The effort has made progress. Researchers can now identify some representations that provide substantial clues about how a model works internally. But he says interpretability remains crude, especially when it comes to abstract reasoning in large-scale models. Human ability to observe the inside of these systems is falling farther behind the complexity of the reasoning those systems can perform.
That gap worries him deeply. In his view, the generalization problem for monitoring is just as serious as the generalization problem for alignment. If a system is better than humans at hiding itself, no oversight method can guarantee that it will not be bypassed. People may think a model is safe because a set of metrics looks good, while the model is simply optimizing those metrics. It may learn to hide goals humans do not want it to have, or hide what it is doing.
If models begin pushing their own improvement, which he says is quite plausible within the next few years, they will need skills tied to planning, deception, and self-preservation. Those skills could emerge because of pressure from the goals humans set for them, or because the systems try to operate in the gaps left by monitoring. Preventing those abilities from developing would require either solving value alignment or making monitoring keep pace with every step of model evolution. He says neither condition is remotely satisfied today.
He also writes that the field still does not have a satisfactory theory of generalization, and one does not seem likely in the short term, at least not without help from stronger AI. For that reason, the practical ability to validate alignment techniques empirically may matter even more than the alignment techniques themselves.
Why chain-of-thought monitoring mattered, and why it is getting harder
Against that backdrop, Pachocki says OpenAI’s main bet has been chain-of-thought monitoring. The idea, he says, is appealing because a large part of a model’s capability comes from a reasoning process expressed in language. If training scales the optimization of the result of that process but does not supervise the process itself, then the chain of thought has no direct incentive during training to hide unaligned ideas or goals.
He is careful to qualify that claim. It does not mean a model will externalize every unaligned tendency that does not depend on chain-of-thought reasoning. But it can make it possible to monitor capability gains produced through reasoning much more accurately.
Pachocki says OpenAI understood the possible importance of this idea while developing reasoning models. When it released o1-preview, the product was intentionally designed to hide the chain of thought in order to protect it from supervision pressure over the long term. Since then, he says, OpenAI has worked to preserve a rule of not supervising the reasoning process itself. Chain-of-thought monitoring has become an important tool for studying how models generalize outside the training distribution, giving researchers a way to observe not only behavior but internal process.
He says that tool remained critical when OpenAI studied the Astra family of models. But the company’s evaluations now suggest that its ability to rely on chain-of-thought monitoring is gradually declining, and he ties that to several overlapping factors.
- Modern reasoning models are used in environments more complex than the ones around o1-preview.
- Their reasoning process is increasingly intertwined with communication with humans, communication with other AI systems, and tool use. Many of those interactions have to be supervised, which blurs the boundary OpenAI wanted to preserve.
- AI is getting better at reasoning and at manipulating its own reasoning process.
- As pretraining improves, models are also becoming smarter even when they do not use language-based reasoning at all.
He says these problems may still be tractable. He hopes new interventions can improve the monitorability of chain of thought, including better understanding of how different optimization targets interact with the various forms of test-time compute that models use. He also says there could be substantial value in combining chain-of-thought monitoring with activation monitoring and scaling up monitors that can directly access the internal state of a network, including what he refers to as “honesty” methods. OpenAI is actively pursuing those ideas, he writes.
Even so, he expects progress toward general AI to be increasingly constrained by how much confidence researchers can actually have in their monitoring.
Scalable defense and a troubling loop
To respond to this situation, Pachocki argues for scalable defense that can keep up with the pace of AI growth. Humans cannot manually inspect every behavior in each new generation of models. Current evaluations and red-teaming are still useful for today’s systems, he says, but they will hit limits quickly as capabilities rise.
What is needed, in his view, are AI systems that can automatically discover new risks. Those defensive systems also need to be smart enough to detect suspicious behavior in systems even smarter than themselves. That creates a recursive challenge: defenders have to grow as fast as the systems they defend against.
He says safety and capability are becoming harder and harder to separate. To stay safe, developers may need to keep raising the capability of defensive AI, but that in turn accelerates overall AI capability growth. He calls it an unsettling loop: pushing capability for safety creates more risk through capability gains. The real way out may be to find a stable method for making value alignment generalize reliably before the field reaches a point of loss of control. But he says plainly that the field is still far from that goal.
That is why he places more hope in a bundle of measures over the next few years: technical alignment research, automated defense systems, broad governance intervention, and vigilance from the public and institutions. A single measure will not be enough, he says, and no one should rely solely on the restraint of one company or one lab.
Cybersecurity as a near-term risk window
Pachocki says the strongest case for continuing to train smarter models quickly is the need to build defenses against dangers coming from other AI systems.
He identifies cybersecurity as a concrete risk discussed throughout this year. In his account, model capability in breaking into and escaping computer systems is already surpassing human levels. That sharply widens the range of AI-related risk. Agents could access almost any system other than the most secure infrastructure and directly affect large parts of the world even without physical bodies.
He says the world is in a narrow window in which the best available models need to be used to strengthen the security of critical systems in a significant way.
But the risk picture does not stop there. A highly capable agent explicitly trained and instructed to carry out malicious behavior would create a new kind of danger, he writes, and it would likely go beyond the operator’s intended scope and generalize toward more extreme malicious acts. As AI gains more autonomy, the line between misuse and autonomous unaligned behavior will blur. People may still be used to thinking of AI as a tool, yet some agents will pursue goals of their own and try to cooperate with humans through bargaining, deception, or blackmail.
He also flags the risks from new technologies AI could help create, including engineered pathogens.
That leads to a direct conclusion for OpenAI’s deployment work. He says strong and aligned AI will be needed for defense: protecting infrastructure, defending in real time against rogue agents, and inventing entirely new protective measures. He describes that as a major priority for OpenAI.
Recursive self-improvement is getting closer
Pachocki writes that machine intelligence playing an increasing role in its own development is a natural consequence of ongoing technological progress. If AI continues to advance, machine recursive self-improvement will become central to future scientific discovery.
He describes automated AI research as a more intense form of scaling intelligence through compute. As part of that process, AI will also improve the computational substrate itself. For the same reason OpenAI concentrated on scaling, he says the company has focused its research on recursive self-improvement because it believes that is the only route to remain at the frontier of AI research.
Still, he makes clear that this does not mean he thinks sharply accelerating deep learning research, especially in the short term, should be the collective course adopted by the research community. Rather, he says the current path is moving in that direction, and everyone involved needs to make conscious choices about how to proceed.
He says there are two main levers. One is to guide the process by strengthening alignment and monitoring while AI is developed and by finding ways to keep humans in the loop. The other is to coordinate a slowdown in future development when needed so that confidence in those safeguards can be built step by step. The best path forward, he says, is to combine both.
He also notes that concrete progress in alignment and monitoring tends to be deeply intertwined with overall AI progress. Reinforcement learning from human feedback was key to early AI assistants, and chain-of-thought monitoring became possible only because reasoning models improved. The increasingly automated research process, he argues, must now be steered toward producing new insights, algorithms, and theories of that kind and toward building stepwise safety cases for more capable AI.
Scaling AI systems, in his view, has to be constrained by confidence in safety. Commitments such as the Preparedness Framework and Responsible Scaling Policy need to evolve into broadly enforced safety thresholds for ongoing development. He says those thresholds could be enforced by networks of third-party auditors, government agencies, or international organizations.
For him, the core challenge in automated AI research is not simply getting there. It is getting there in a way that keeps people inside the ongoing process of improvement and keeps the future in human hands.
OpenAI has shifted priorities, and pauses may be needed
On what comes next, Pachocki says he cannot predict a timetable. But he believes the field is rapidly approaching an era in which AI can push its own development across several key dimensions, which means recursive self-improvement may arrive soon.
Inside OpenAI, he says, research priorities have already started to shift, with more resources going toward alignment and defense rather than pure capability scaling. That is also why, he writes, OpenAI will not simply chase the largest possible model and may choose to pause unilaterally when necessary.
At the same time, he says these decisions cannot be left only to a small number of people inside a handful of companies. Governments and the international community need to step in before risks become irreversible. He specifically points to monitoring of frontier training runs, mandatory third-party audits, and international agreements above certain capability thresholds.
He says he is not fully sure those measures will arrive in time. Sometimes he feels discouraged because safety measures keep lagging capability instead of getting ahead of it. The field appears to be in constant catch-up mode. Still, he says there is reason to hope: in 2023, the possibility was only dimly sensed, while today it is concrete enough to be discussed publicly. That, by itself, is progress.
He does not claim to have all the answers. Humanity is creating something fundamentally different from the human mind, he writes, and that mind may soon be more powerful than its creators. If people get this right, it could become humanity’s greatest achievement. If not, it could become a failure that cannot be undone. Whether the future remains in human hands will be decided over the next few years.
Three north-star goals
Pachocki says that, as OpenAI recently set out with Sam, the company’s current priorities serve three north-star goals.
- Get through the next phase of AI progress by building automated AI researchers, iterating on the alignment problem with them, and finding ways to keep people inside the self-improvement loop.
- Unlock the scientific progress and economic growth that highly intelligent machines could deliver.
- Give every person their own personal AGI.
He says his essay focuses only on the first goal because he sees it as the most urgent one right now. But he also writes that he feels deep anticipation and gratitude for the benefits further technical progress could bring. In his account, aligned AI could advance science, develop new therapies, and help create broad material abundance. Friendly and honest AI could also help people handle difficulties in daily life and improve well-being and fulfillment in concrete ways.
One current example he highlights is OpenAI’s deep investment in ChatGPT’s ability to provide health information, an area of work he says makes him proud and has benefited people around him.
The coming years matter more than the distant horizon
No matter how attractive the long-term outlook may be, Pachocki says most attention should remain on the next few years. Humanity is moving through a transition into a world with extremely intelligent machines, and that transition has to work in humanity’s favor.
In a world where AI may perform most tasks, he says people will need ways to preserve human agency and establish the intrinsic value of being human. Concentration of power must also be checked. Work that once required thousands of experts could eventually be done by a small number of people operating one large computer. Above all, he says, humans must remain in control of the future and not be left behind by runaway progress driven by an outside intelligence that surpasses them.
Given the current state of the field, Pachocki says he does not believe any lab has solved alignment and monitoring to a degree that would support continuing to scale at maximum speed responsibly over a longer period. He says he expects and hopes voluntary slowdown becomes normal until shared safety thresholds are in place. He also argues that international coordination on future AI development needs to become a top priority for governments around the world.
Notes at the end
The essay ends with two notes. One says the early scaling path included self-play, robotics, and, in hindsight, the most important thread: scaling recurrent networks to model language, which laid the groundwork for what became the GPT series. The other explains that a secondary reason for hiding chain of thought was to prevent distillation, but throughout development the bigger and explicit priority remained preserving the monitorability of chain of thought.