Nick Cammarata, one of the people who helped start OpenAI’s interpretability team, put out a long essay this week saying modern agent tooling has made even trillion-parameter models "almost readable." His basic point is blunt: one of AI’s oldest assumptions—that neural networks are opaque black boxes—may have been exaggerated, and a lot of the barriers researchers braced for might not actually be there.
The essay is billed as a 65-minute read. And people are already passing it around, trying to sort out what it means. The takeaway getting the most attention is simple: in Qwen3-32B, about 90% of neurons may mostly be doing one main thing.
A central interpretability assumption is being challenged
For years, a pretty grim assumption sat underneath interpretability research: neural networks were supposed to be deeply tangled machines. If neurons had limited capacity, the theory said, models would cram unrelated concepts into the same neuron whenever those concepts didn’t clash in practice. That idea—usually called superposition—became a base layer for a lot of AI safety work.
Under that view, researchers often had to train a second model, like a sparse autoencoder, to pull apart those representations before they could even start figuring out what the original network was doing.
Cammarata now says that whole picture may be off. In his telling, the network may have been far cleaner than people thought, and the mess may have come from how humans read the evidence in front of them.
The old "cat face and car front" example may have been misread
Mechanistic interpretability is an attempt to trace neurons and circuits so researchers can explain how a model gets to an answer. One old problem keeps coming up: a single neuron can fire for things that seem totally unrelated.
A famous example came from the vision model InceptionV1. The same neuron was said to detect both cat faces and car fronts. The standard reading was that the model had stuffed unrelated concepts into one representation because it was short on space.
But after going back through the images and the connection structure, Cammarata says the example may mean something else. Cat faces and car fronts may share a lower-level feature: a V-shape. The same neuron also seems connected to V-shaped flames and fox faces, while round shapes drive it toward negative activation.
So on that reading, "cat" and "car" are human labels. The network may really be picking up a shape pattern that runs across categories. Cammarata uses that example to argue that neurons are not always object labels in noun form. Sometimes they may work more like operators—splitting patterns apart and sending information into different downstream circuits.
Polysemy may not mean random concept packing
Cammarata is not claiming every neuron does only one thing. That’s not his argument. His point is that even when a neuron seems multi-purpose, that still may not require the strongest version of the superposition story.
He suggests what he calls recursive generalization. A neuron starts with one pattern, then spreads into nearby branches, and those branches keep growing. In examples from the essay, a neuron that recognizes "iy" can extend to "diy," then to "do it yourself," even when the full phrase no longer literally contains "iy." A geology-related neuron can branch toward Cesium, the geographic JavaScript library.
He says that looks less like unrelated information being hidden in some high-dimensional space and more like expansion across nearby semantic territory. And he goes a step further: superposition is the most aggressive and hardest-to-falsify explanation, so researchers should treat it as a last guess, not the first one. He also says he believed it himself for years.
If researchers look only at final activations, those links can seem weak, weird, even arbitrary. But if they follow the training path, he argues, the route connecting them becomes easier to spot.
Interpretation tools can mislead as well
The essay also says automated interpretation systems can introduce their own errors. One neuron that an automatic tool labeled a "polysemous comma neuron" turned out, according to the team, to be tied to Colorado. It reacted to local phone area codes and slang, and boosting it made the model more likely to plan hiking trips.
In that example, the comma was just the place where the model pulled together the earlier context. Looking at the character where the activation showed up risked missing the real meaning entirely.
Preliminary review of Qwen3-32B: about 90% of neurons have one main job on each pole
Cammarata sets these results against much older ones. Back in 2023, OpenAI’s automated interpretability project found only about 1.7% interpretable neurons, even using its loosest bar.
Now, in his preliminary review of Qwen3-32B, he estimates that about 90% of neurons have one main job on each positive and negative pole, around 7% have two related jobs, and roughly 2% to 4% are still not understood. But he also says those figures shift depending on the activation threshold and on what exactly counts as "one job."
He says the same broad pattern shows up in dense models and in mixture-of-experts models. On Kimi K3, the team produced what he calls the first attribution map for a trillion-parameter model. In one example built around the question of finding a lost wallet, wallet, cash, and identification connect to lost-and-found and honesty, and the final output becomes "return it."
And when the newer tools were used to revisit GPT-2 XL, he estimates that about 70% of neurons there also look roughly like "one thing."
Agents changed the scale of the research
A second big thread in the essay is scale. Cammarata says AI systems can inspect neuron samples, suggest explanations, trace upstream and downstream connections, and set up experiments that hunt for counterexamples. Researchers used agents to inspect 100,000 attribution graphs. He says he personally reviewed several thousand more.
The essay also takes on the obvious objection: how do you keep an AI system from spinning a story that just sounds convincing? One test described in the piece uses two steps. First, one agent studies a neuron and writes an explanation. Then a second agent, given only that explanation, creates a new sample that should activate the neuron strongly. That sample is then checked against the network itself.
Cammarata says some of those generated samples activated neurons more strongly than any examples found in a corpus containing tens of billions of tokens. The team also made near-miss counterexamples that looked similar but were not supposed to trigger the neuron. Then they used interventions like neuron removal and activation boosting to test whether the interpretation really predicted model behavior.
Under that setup, an explanation stops being a polished story and becomes a hypothesis that can fail. Better.
More understanding can make the rest easier to read
Cammarata also makes a point that sounds backward at first: understanding more of a network can make the leftover parts easier to understand too. He compares it to a crossword puzzle. Fill in more of the nearby answers, and the possible choices for a blank tighten fast.
He also says larger models may have more specialized neurons. If that is right, they could actually be easier to identify, not harder.
That leads him to what he calls a kind of bitter lesson for interpretability itself: once the scale is big enough, plain observation, hypothesis-building, and testing may work better than repeatedly inventing elaborate decomposition methods.
He does not say interpretability is solved
The essay does not declare victory. Cammarata says there is still a lot of work left on attention mechanisms, full-model auditing, and the problem of turning understanding into dependable safety gains.
Still, the direction he sketches is concrete. If neural networks already contain internal structure that can be traced, then future progress may come from a lab where AI scientists help run nonstop experiments, inspect models, ask questions, and test answers one by one.
The source article also points to reference material from already-clean.goodfire.com and to a related discussion on Hacker News. It says the original Chinese article came from the WeChat public account "Heart of the Machine," with the author credited as "Focused on Large Models."