A team at the University of Tokyo has introduced MIDA, or Multimodal Interactive Deception Assessment, to test how well multimodal large language models handle deception in multi-person social interaction. The study reports that even stronger multimodal systems still struggle to reliably separate truthful statements from lies. The paper has been accepted to the EMNLP 2026 Main Conference.

The authors are Caixin Kang, Yifei Huang, Liangyang Ouyang, Mingfang Zhang, Ruicong Liu and Yoichi Sato, all from the University of Tokyo. Their work focuses on multimodal understanding, video analysis and social intelligence, with this paper centered on deception judgment and theory-of-mind reasoning in complex group interaction. The paper, project page, code repository and dataset page have all been released publicly.
Why the team used Werewolf as a testbed
Earlier deception-detection research has mostly focused on single-speaker statements or two-person dialogue. In real settings, though, deception rarely appears in isolation. It is mixed into questioning, alliance building, defense and group pressure, and it shifts as new information appears. Another problem is that real-world speech often lacks a fully reliable ground truth for whether a statement is a lie.
The researchers chose One Night Ultimate Werewolf because the game addresses both issues at once. Each player has a hidden role, night actions determine what information that player knows, and daytime discussion reflects judgment and strategy. Because roles, actions and rules are traceable, the researchers can determine whether a statement conflicts with what the speaker actually knew at that moment.

That means MIDA is not asking whether someone merely looks deceptive. It asks what the person knew and what the person said.
How the MIDA dataset is built
MIDA contains 2,360 utterances in total. Of those, 819 come from 40 in-person games in Ego4D, while 1,541 come from 151 YouTube game videos. The two subsets create a natural difficulty gradient: participants in Ego4D are mostly beginners, while players in the YouTube videos are more experienced and more likely to use complex deceptive strategies.
Each sample includes video, audio, dialogue history and game rules, and each is labeled as TRUE, FALSE or NEUTRAL. Here, NEUTRAL does not mean emotionally neutral. It refers to statements that do not contain a fact that can be verified against the speaker’s current knowledge state, such as questions, guesses, opinions or hypotheses.
To obtain verifiable labels, the team first manually recorded the night actions in each game, then combined those records with roles, rules and dialogue content to generate and validate labels for every utterance. The paper notes that roles, night actions and voting outcomes are used only to construct ground-truth labels. They are not given to the models during testing. The models still have to infer what the speaker knows and what the speaker is trying to do from video, audio and dialogue history alone.

Beyond truthfulness, MIDA also asks models to identify the persuasion strategy used in each utterance, including identity claims, accusations, defense, evidence, questioning and calls to action.
Why overall accuracy can give the wrong impression
The evaluation covers GPT-4o, GPT-5, Gemini, Claude, as well as Llama, InternVL, Qwen and DeepSeek.
At the level of overall accuracy, some models appear to perform reasonably well. On the Ego4D subset, GPT-4o reaches 74.0% overall accuracy and Qwen2.5-VL-7B reaches 76.4%. But MIDA contains a large number of neutral utterances, so a model can post a decent-looking score simply by choosing NEUTRAL often.
Once the researchers keep only the samples that must be judged true or false, the picture changes sharply. GPT-4o records binary accuracy of just 27.6%, while GPT-4o-mini reaches 39.4%. Qwen2.5-VL-7B has the highest overall accuracy, yet its F1 score on FALSE is 0.

In other words, a model may answer “cannot determine” most of the time, earn a relatively high overall score, and still fail to identify lies in any meaningful sense.
The weakest results appear in the categories that require the most social reasoning. For identity-claim utterances, the average accuracy across models is only 15.7% on one subset and 25.3% on the other. For evidence-based utterances, average accuracy stays below 40% in both cases. A model may recognize that a statement is making an accusation, but that does not mean it can tell whether the accusation is actually possible.
Experienced players make the task harder still. In the YouTube subset, the share of false evidence reaches 36.8%, close to three times the level seen among beginners. The share of identity lies is 51.6%, well above the 18.9% reported for novice players.
Dialogue history matters more than extra visual detail
The team ran ablation studies to measure the contribution of different inputs. When prior dialogue history is removed, GPT-4o-mini’s accuracy on true-or-false samples falls from 39.4% to 13.4%.

That result suggests a split between local and social understanding. A model may only need local text to tell whether a sentence is a question, a defense or an accusation. But to judge whether it is true, the model has to remember who said what, who knows what, and whether different players’ statements support or contradict one another.
More visual information does not automatically help. Adding one frame raises macro F1 by 2.5 percentage points. Expanding that to three frames reduces the gain to 2.1 points. More frames bring more facial expressions and body movements, but they also add more noise. The model may see more cues without knowing which ones actually matter.
SoCoT and DSEM take different approaches
To address these issues, the researchers designed two modules: Social Chain-of-Thought, or SoCoT, and Dynamic Social Episodic Memory, or DSEM.
SoCoT asks the model to first describe facial expression, body posture and vocal traits, then infer the speaker’s goal and mental state, and only after that make a truthfulness judgment. The idea is to break an intuition such as “the speaker’s voice is shaking, so something may be off” into reasoning steps that can be inspected.

But nonverbal cues do not always improve judgment. In GPT-4o-mini, analyzing facial cues alone raises overall accuracy by 8 percentage points, yet binary accuracy on true-versus-false statements drops by 7.1 points. The paper argues that behaviors that look suspicious, such as nervousness or avoiding eye contact, are not the same thing as lying. A model may detect those behaviors and still interpret them incorrectly.
DSEM works differently. It maintains a continuously updated memory board for each player, recording role claims, actions, observations, known facts, interpersonal relations and the event timeline. When a new utterance appears, the model does not focus only on the current frame. It checks what that person should know at that point, whether the statement conflicts with earlier information, and whether other players’ remarks support it.
In the experiments, DSEM produces more stable gains. It improves GPT-4o-mini’s binary accuracy by 2.3 percentage points and macro F1 by 3.3 points. On Qwen2.5-VL-7B, it lifts macro precision by 7.6 points. Compared with searching for a tell in a single moment, tracking each participant’s knowledge and stance over time appears closer to how humans actually reason in social settings.
What the paper says this means
The authors argue that MIDA is not only about Werewolf. In meeting collaboration, online discussion, human-computer interaction, intelligent customer service and multi-agent collaboration, models also need to track what different participants know, how their positions change and how they affect one another.

In the paper’s framing, current multimodal large models are powerful “knowledge engines,” but not yet mature “social agents.” What they lack may not simply be a stronger visual encoder. The missing piece may be the ability to continuously model other people’s knowledge, beliefs and intentions, which is commonly described as theory of mind. Once a model leaves visual question answering with clear answers and enters an environment where participants hold different private information, views keep shifting and people actively try to influence one another, even today’s strongest multimodal systems still have trouble truly reading the room.
Public materials include the paper at https://arxiv.org/abs/2511.16221, the project page at https://ut-vision-social.github.io/, the code repository at https://github.com/kkkcx/Can-MLLMs-Read-the-Room and the dataset at https://huggingface.co/datasets/Ukpkmkkk/MIDA.
The article credits the WeChat public account Xinzhiyuan, with authorship listed as Xinzhiyuan.


