A Tsinghua University research team said object hallucinations in multimodal large language models are not driven only by language priors. In short-answer settings such as Yes/No responses, the model leans more on visual inputs, and a different hallucination mechanism emerges from visual feature extraction itself. The paper names that mechanism “visual-origin hallucination” and proposes ACFT as a remedy.
The work appears in a paper for ACM MM 2026, the 34th ACM International Conference on Multimedia. The paper is available at http://arxiv.org/abs/2609.00231, and the code is posted at https://github.com/zxp555/ACFT_MM26.
The authors said ACFT uses only 0.9% of the COCO dataset and adds no inference cost, while delivering strong results on POPE, MME, and four description-level hallucination benchmarks across LLaVA, MiniGPT-4, and Qwen2.5-VL.
Why the team looked beyond language priors
Object hallucination remains a central reliability problem for multimodal large language models. A model may report objects that are not present in an image, or fail to recognize objects that are there. In high-risk applications such as autonomous driving and medical assistance, those errors can be costly.
Most prior explanations have centered on language priors. Under that view, models repeatedly exposed to co-occurring concepts in training text may answer according to statistical regularities even when the image does not support the claim. Earlier work has linked the issue to overreliance on text co-occurrence statistics, hallucination-focused attention heads that concentrate on text tokens, attention aggregation around summary tokens, or insufficient supervision for fine-grained reasoning.
Mitigation methods have largely followed the same assumption. VCD and OPERA intervene during decoding at the input level. Woodpecker applies output-side post-processing with an external grounding module. RLHF and DPO belong to the family of post-training alignment methods.
The Tsinghua team argues that these approaches share a common premise: hallucinations mainly come from text-side bias. That premise can hold in long-form outputs, such as detailed image descriptions, because richer context amplifies language bias. But when the output collapses to a short “yes” or “no,” the role of language priors weakens sharply, and the effectiveness of those methods also drops.
That led the researchers to ask where hallucinations come from in short-output settings.
Two analyses pointed to visual misalignment
To test whether visual-origin hallucination exists, the researchers ran two complementary analyses on LLaVA v1.5 and reported quantitative evidence.
The first finding was image-text embedding misalignment. The cosine similarity between image and text embeddings was much lower for hallucinated samples than for correct ones. Correct samples averaged 0.158, while hallucinated samples averaged -0.122, which the paper describes as a systematic collapse in cross-modal alignment.
The second finding was a reversal in attention patterns. Using Smooth Grad-CAM, the team visualized model attention and defined a semantically reasonable pattern: when the target object exists, attention should concentrate on the target region; when the target object does not exist, attention should be dispersed.
Across 500 hallucinated samples and 500 non-hallucinated samples, normalized Shannon entropy showed a consistent violation of that pattern. When the object was present, entropy was 5.1% higher, indicating overly diffuse attention and missed targets. When the object was absent, entropy was 6.2% lower, indicating mistaken concentration on irrelevant regions that triggered hallucination.
The team also carried out a causal test rather than stopping at correlation. Researchers directly intervened on the visual encoder by adding Gaussian noise, downsampling, and swapping in weaker encoders. Under those changes, average POPE accuracy fell from 0.842 to a range of 0.739 to 0.822. Replacing the encoder with the stronger SigLIP-SO400M pushed accuracy up to 0.864. The paper presents this as causal support for the claim that visual feature quality drives object-existence hallucinations.
AHAF flips hallucination attributes with adversarial perturbations
Once the diagnosis pointed to visual misalignment, contrastive learning became the most direct correction path. But the researchers found that ordinary contrastive fine-tuning, or OCFT, did not work well. In OCFT, a matched image is used as a positive sample and a random unrelated image is used as a negative sample.
According to the paper, the problem is that the feature gap between positive and negative samples is uncontrolled and not centered on the target object. That makes it hard for the model to learn which visual features actually trigger hallucination.
The team therefore proposed AHAF, short for Adversarial Hallucination Attribute Flipping. The method uses PGD to apply directed adversarial perturbations within a very small ℓ∞ ball, turning an image that does not trigger hallucination into one that does. The resulting positive and negative pair is fully aligned except for the controlled perturbation.
AHAF also serves as a diagnostic probe. The authors said that if a tiny pixel-level perturbation can flip the model’s answer, then the visual representation of the multimodal model is already sitting dangerously close to the hallucination decision boundary even on clean images. In the paper, that behavior reinforces the diagnosis of visual-origin hallucination.
ACFT trains on aligned pairs and leaves inference unchanged
Built on the aligned pairs generated by AHAF, ACFT — Adversarial Contrastive Fine-Tuning — optimizes the embedding space by maximizing similarity between a text anchor and the positive image while minimizing similarity between the same anchor and the negative image.
The paper highlights three engineering properties of the method:
- It does not depend on a specific backbone architecture.
- It requires only a small amount of data.
- Everything happens during training, with zero extra cost at inference time.
Evaluation covered LLaVA, MiniGPT-4, and Qwen2.5-VL
The researchers evaluated LLaVA v1.5-7B, MiniGPT-4 13B, and Qwen2.5-VL-7B on POPE and MME. The paper said those two benchmarks were chosen because all questions are answered in Yes/No form, matching the short-output setting at the center of the study.
In the main POPE experiment, ACFT reached accuracies of 0.841, 0.906, and 0.897 on the three LLaVA subsets, beating the second-best baseline by 3.3%, 2.0%, and 0.5%, respectively. On MiniGPT-4, the gains over the second-best baseline were 3.0%, 5.3%, and 2.5%. On Qwen2.5-VL, where the baseline was already strong, accuracy still improved from 0.864, 0.875, and 0.884 to 0.877, 0.900, and 0.916.
Ablation results highlighted the value of aligned sample pairs
One of the paper’s clearest comparisons used the same 3,000 COCO images to train OCFT and ACFT separately.
ACFT outperformed OCFT by 35.8%, 7.4%, and 17.6% on the three subsets. The gap was especially striking on the Adversarial subset, where OCFT reached only 0.483 accuracy, even below the untuned original LLaVA model. The authors said this suggests unaligned negative samples are not just unhelpful but harmful.
A further similarity-gap analysis offered an explanation. Under ACFT, the similarity difference between positive and negative samples for the target object, truck, was clearly distinct from the pattern for non-target objects such as dog, cat, and table. OCFT did not show that property. The paper argues that ACFT teaches the model a more consistent rule: focus on differences in the target object’s own features.
Visualization and entropy both moved in the expected direction
Visualization results showed that ACFT corrected the two symptoms identified earlier. Image-text embedding cosine similarity increased noticeably, and Grad-CAM attention returned to a semantically reasonable distribution: concentrated when the object exists, dispersed when it does not.
Entropy analysis pointed the same way. In cases where the object existed, entropy fell by 8.7% and 2.6%. In cases where the object did not exist, entropy rose by 7.4% and 6.4%.
Authors and paper details
The paper lists Xu Peiyang, Zhu Xiaopei, Zhu Jun, and Hu Xiaolin as authors. Xu Peiyang is a co-first author and an undergraduate student at Tsinghua University. Zhu Xiaopei is also a co-first author and a Shuimu Scholar at Tsinghua University, working with Professor Zhu Jun. Professor Zhu Jun and Associate Professor Hu Xiaolin are the corresponding authors.
The reference link is http://arxiv.org/abs/2609.00231. The original article cited in the input says the piece came from the WeChat public account “Xinzhiyuan,” written by Xinzhiyuan and edited by LRST.
Beyond the existing language-prior explanation, the paper identifies and characterizes a hallucination mechanism driven by errors in visual feature extraction and image-text embedding misalignment. It also presents AHAF and ACFT together: AHAF acts both as a probe for fragile visual representations in multimodal models and as a way to generate aligned contrastive training data, while ACFT is framed as a data-efficient fine-tuning method that uses 0.9% of COCO and adds no inference overhead.