Google Research says GPT-5, Gemini 3 and other language models often fail factual questions not because the facts are absent from their parameters, but because the models cannot retrieve them on demand.
The study, titled Empty Shelves or Lost Keys?, was accepted at ICML 2026. On Aug. 12, Google Research published a blog post summarizing the results and released a new benchmark and full dataset called WikiProfile.
From missing knowledge to failed recall
The paper separates factual mistakes into two categories. One is a storage problem: the model never encoded the fact in pretraining. The other is a recall problem: the fact is present in parameters, but the model cannot surface it when asked.
Google argues that these two failure modes have often been collapsed into a single accuracy number even though they call for different fixes. The study introduces a "knowledge states" framework that asks what state a fact is in inside a model rather than whether one answer is simply right or wrong.
The framework uses five states: encoding failure, recall failure, direct recall, thinking-mediated recall, and correct reasoning without prior storage.
How WikiProfile was built
To test whether a fact had been stored, the researchers truncated a Wikipedia passage right before the answer appeared and asked the model to continue it, or asked a question in the same context. Thinking was disabled in these tests to separate stored knowledge from on-the-fly inference.
WikiProfile contains 2,150 facts, all drawn from English Wikipedia. Each fact comes with 10 questions: 2 for storage, 4 free-response recall questions, and 4 multiple-choice recognition questions, for a total of 21,500 prompts.
Question generation was handled by Gemini-2.5-Pro, then checked one by one with Google Search. If an answer was not unique or a question was ambiguous, the entire fact entry was discarded. A final human review removed fewer than 2% more.
Google evaluated 13 models from four families: Gemini 3, GPT-5, GPT-4.1 and Gemma 3. Each model was tested with thinking on and off. Every question was sampled eight times, producing about 4.5 million responses in total. The paper says a full evaluation run for one frontier model costs about $500.
Most facts were stored, many still could not be produced
The headline result is that frontier models stored 95%–98% of benchmark facts in their parameters, but still failed to answer 26%–34% of those facts in direct questioning. With thinking enabled, 11%–12% still could not be recalled.
That shifts the bottleneck. In this benchmark, factual failure at the frontier appears to be dominated less by missing knowledge and more by inaccessible knowledge.
Rare facts are stored more often than they are recalled
One major choke point was rare knowledge. A common explanation has been that long-tail facts are missing because model capacity is limited. Google’s results point elsewhere.
For Gemini-3-Pro, the model stored 94.5% of rare facts and 99.5% of popular facts, only a 5-point gap. In direct recall, though, rare facts dropped to 63.3% while popular facts stayed at 84.7%, a 21-point spread.
Storage was close. Retrieval was not.
The gap was larger for GPT-5. The study reports storage rates of 90.7% for rare facts and 98.4% for popular ones. But in free-response answering, rare facts fell to 52.9% while popular facts remained at 77.8%.
The paper’s interpretation is straightforward: many long-tail facts are in the model, but they are harder to wake up.
Reversed questions expose another retrieval bottleneck
The second recurring failure mode showed up in reversed questions. A model may answer "Who is Tom Cruise’s mother?" quickly and correctly, then stumble when asked the reverse, such as who that person’s son is.
A 2023 paper described this as the "reversal curse," with one explanation being that the model learned "A is B" but did not internalize the reverse relation "B is A." Google tested that idea by converting the same items into four-option multiple-choice questions.
For GPT-5, direct recall was 82.9% on forward questions and 74% on reverse ones. But once those same facts were turned into recognition tasks, reverse questions were no harder than forward questions. Across the 13 models, 9 did better on reverse recognition and the other 4 were tied.
That matters. The models could recognize the answer when it was present, but often could not generate it from scratch. Google says this suggests the association itself is there; the problem is surfacing it.
The authors argue that facts are stored together with training-time context, wording and order. When a prompt departs from that familiar form, retrieval gets harder.
Thinking may act as a recall tool
Most WikiProfile questions are simple factual prompts without a multi-step reasoning path. A question like where Oasis played its first show is either remembered or not, so thinking would not obviously be expected to help much.
Yet it did. For facts that had been stored but could not be recalled directly, thinking recovered 40%–65%. For facts that had never been stored, thinking recovered only 5%–15%.
Google takes that as evidence that thinking is often not just a reasoning aid. It can also help the model search its own parametric memory.
The paper also tested a simpler explanation: maybe thinking merely makes answers longer and increases the odds that one of eight samples lands on the correct output by chance. If that were true, answers should become more scattered. The opposite happened. With thinking enabled, the number of correct responses across the eight samples became more concentrated for the same question.
The largest gains showed up where recall was weakest. For Gemini-3-Pro, the gap between rare and popular facts narrowed from 21.4 points to 12.5 points. For GPT-5, the gap between forward and reverse questions narrowed from 9 points to 2 points.
Google’s related paper, Thinking to Recall, offers two mechanisms. One is a compute-buffer effect, where the extra tokens generated during thinking provide additional implicit computation. The other is fact priming, where the model first activates related facts and then reaches the target answer through a semantic bridge. The paper compares that process to spreading activation in human cognition.
It also warns about the downside. If the intermediate facts inside a chain of thought are wrong, the final hallucination risk can rise instead of fall.
Scaling fills shelves, but does not necessarily find the key
The study uses the open-source Gemma 3 family as a scale reference. As parameter size increased from 1B to 27B, encoding failure fell from 85% to 23%.
That is a clear storage improvement. Recall failure, however, did not fall in step. Its share of the remaining errors kept climbing and became the dominant error type at 27B.
The pattern was even sharper in frontier models. The paper says recall failure accounted for more than 70% of all errors in GPT-5.2, and that the share rises as models get stronger. Across all 13 models, encoding failure shrank with scale while recall failure changed little.
Google’s conclusion is not that scaling stops helping. It is that scaling mostly improves how much knowledge gets stored, not how much can be retrieved reliably once stored. If parametric storage is already reaching a 95%–98% ceiling on this benchmark, then adding more parameters and more data may deliver smaller gains in factual accuracy than before.
The paper notes that retrieval-augmented methods can still fill part of the gap. At the same time, knowledge inside parameters still matters for fluency, response speed and integration across contexts, so external retrieval is not a complete substitute.
Next question: can a model tell when it cannot recall?
The paper frames the next step as a metacognition problem. A model first has to recognize that it cannot answer directly before it can decide a question is worth spending extra thinking on.
For the last two years, the default recipe for more reliable models has been more data and larger parameter counts. Google’s study draws a line under that approach in factual recall: frontier systems may already have stocked most of the shelf. The harder part now is getting the right item back out.
And after that comes an even tougher requirement. The model has to know that it is not ignorant of the answer, only unable to remember it right away.
The references cited in the source include Google Research’s blog post, "Empty Shelves or Lost Keys? Recall is the Bottleneck for Parametric Factuality," and a related item on aihot.virxact.com. The original Chinese article was published by the WeChat account Xinzhiyuan and credited to ASI Qishilu.