Accepting an entire financial report, contract or research paper is not the same as using its information correctly. A paper accepted by EMNLP 2026, titled “Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Models,” examines the internal process that allows large models to locate relevant evidence in long documents and identifies a class of attention heads involved in multimodal retrieval.
Long-context vision-language models can process hundreds of pages containing text, images, tables and charts. The answer, though, usually depends on only a small part of that input. The relevant evidence may be a passage of text, an item in a table, an image or a specific area of a page.
Moving from storing context to finding evidence
The research team asked whether the model contains a set of attention heads that directs a question toward the text or visual content needed to answer it once the evidence is already present in the input.
The team calls the identified heads multimodal retrieval heads, or MMRetHeads.
How MMRetHeads are detected
Inspired by QRHead, the team proposed a method for detecting MMRetHeads. It examines each attention head’s question-to-evidence attention: the attention from question tokens to annotated evidence regions.
For text evidence, the target region consists of text tokens. For evidence located inside an image, it consists of visual tokens. The stronger the attention directed toward the evidence, the higher the retrieval score assigned to that head.
The study covers six long-context vision-language models, including Qwen3-VL and Gemma3. Its evaluation includes text retrieval, image retrieval, rendered-text retrieval and same-image retrieval. The tested context lengths are 8K, 16K, 32K, 64K and 128K.
The analysis found that text and image retrieval share some attention heads. Different context lengths and different forms of evidence also activate different attention heads.
Masking the heads causes a sharp performance drop
Attention directed toward evidence establishes an association, but it does not by itself show that the model depends on those heads. To test that question, the team masked high-scoring retrieval heads and compared the results with those from masking the same number of randomly selected attention heads.
Across text and image retrieval tasks, masking the retrieval heads caused a significant performance decline under different context lengths and evidence positions. Random masking had a much smaller effect.
The same pattern appeared in more realistic long-document question-answering tasks:
- MMLongBench-Doc: 48.2 to 5.7;
- SlideVQA: 71.2 to 8.9.
After random masking, the two tasks still retained scores of 32.2 and 52.6, respectively. The much larger decline caused by masking the retrieval heads indicates that they do more than attend near the evidence. The results point to a causal role in the model’s access to that evidence.
A similar result appeared in multimodal reasoning tasks. After the retrieval heads were masked, the model sometimes answered that there was not enough information even though the chart remained in the input. In other cases, it read the wrong content or generated supporting evidence that did not exist in the input.
Using internal signals for document retrieval
The team then used evidence signals from the attention heads for multimodal document retrieval. Given a question, MMRetHeads calculates relevance scores for candidate pages or layout regions and ranks the candidates by those scores. The process does not require an additional retriever to be trained.
Page-level retrieval searches for the full page containing the evidence. Layout-level retrieval goes a step further, locating text blocks, tables, images or charts within the page.
On MMDocIR, Qwen3-VL-8B achieved a page-level Recall@1 of 64.7, 7.7 percentage points above the strongest reported baseline. Its layout-level Recall@1 reached 39.0, 6.3 percentage points above that baseline.
Paper: https://arxiv.org/abs/2605.27243
This article was originally published by the WeChat account “QbitAI” and written by the MMRetHeads team.