University of Tokyo’s MIDA benchmark finds top multimodal models still struggle to detect deception in group play

University of Tokyo’s MIDA benchmark finds top multimodal models still struggle to detect deception in group play

N
News Editor
2026-10-29 03:57:16
A research team from the University of Tokyo has introduced MIDA, short for Multimodal Interactive Deception Assessment, a benchmark designed to test whether multimodal large language models can judge deception in complex multi-person interactions. The paper has been accepted to the EMNLP 2026 Main Conference and evaluates both closed-source and open-source systems including GPT-4o, GPT-5, Gemini, Claude, Llama, InternVL, Qwen and DeepSeek. MIDA is built on games of One Night Ultimate Werewolf, where hidden roles, night actions and public discussion create a setting in which statements can be checked against what each speaker actually knew at the time. The dataset contains 2,360 utterances, including 819 from 40 in-person Ego4D games and 1,541 from 151 YouTube game videos. Each sample includes video, audio, dialogue history and game rules, and is labeled TRUE, FALSE or NEUTRAL. The study finds that headline accuracy can be misleading because many utterances are neutral. Once evaluation is restricted to statements that must be judged true or false, model performance drops sharply. The paper also reports that dialogue history matters more than adding extra visual frames, while a dynamic memory method called DSEM delivers more stable gains than relying on nonverbal cues alone.

A team at the University of Tokyo has introduced MIDA, or Multimodal Interactive Deception Assessment, to test how well multimodal large language models handle deception in multi-person social interaction. The study reports that even stronger multimodal systems still struggle to reliably separate truthful statements from lies. The paper has been accepted to the EMNLP 2026 Main Conference.

University of Tokyo’s MIDA benchmark finds top multimodal models still struggle to detect deception in group play 2

The authors are Caixin Kang, Yifei Huang, Liangyang Ouyang, Mingfang Zhang, Ruicong Liu and Yoichi Sato, all from the University of Tokyo. Their work focuses on multimodal understanding, video analysis and social intelligence, with this paper centered on deception judgment and theory-of-mind reasoning in complex group interaction. The paper, project page, code repository and dataset page have all been released publicly.

Why the team used Werewolf as a testbed

Earlier deception-detection research has mostly focused on single-speaker statements or two-person dialogue. In real settings, though, deception rarely appears in isolation. It is mixed into questioning, alliance building, defense and group pressure, and it shifts as new information appears. Another problem is that real-world speech often lacks a fully reliable ground truth for whether a statement is a lie.

The researchers chose One Night Ultimate Werewolf because the game addresses both issues at once. Each player has a hidden role, night actions determine what information that player knows, and daytime discussion reflects judgment and strategy. Because roles, actions and rules are traceable, the researchers can determine whether a statement conflicts with what the speaker actually knew at that moment.

University of Tokyo’s MIDA benchmark finds top multimodal models still struggle to detect deception in group play 3

That means MIDA is not asking whether someone merely looks deceptive. It asks what the person knew and what the person said.

How the MIDA dataset is built

MIDA contains 2,360 utterances in total. Of those, 819 come from 40 in-person games in Ego4D, while 1,541 come from 151 YouTube game videos. The two subsets create a natural difficulty gradient: participants in Ego4D are mostly beginners, while players in the YouTube videos are more experienced and more likely to use complex deceptive strategies.

Each sample includes video, audio, dialogue history and game rules, and each is labeled as TRUE, FALSE or NEUTRAL. Here, NEUTRAL does not mean emotionally neutral. It refers to statements that do not contain a fact that can be verified against the speaker’s current knowledge state, such as questions, guesses, opinions or hypotheses.

To obtain verifiable labels, the team first manually recorded the night actions in each game, then combined those records with roles, rules and dialogue content to generate and validate labels for every utterance. The paper notes that roles, night actions and voting outcomes are used only to construct ground-truth labels. They are not given to the models during testing. The models still have to infer what the speaker knows and what the speaker is trying to do from video, audio and dialogue history alone.

University of Tokyo’s MIDA benchmark finds top multimodal models still struggle to detect deception in group play 4

Beyond truthfulness, MIDA also asks models to identify the persuasion strategy used in each utterance, including identity claims, accusations, defense, evidence, questioning and calls to action.

Why overall accuracy can give the wrong impression

The evaluation covers GPT-4o, GPT-5, Gemini, Claude, as well as Llama, InternVL, Qwen and DeepSeek.

At the level of overall accuracy, some models appear to perform reasonably well. On the Ego4D subset, GPT-4o reaches 74.0% overall accuracy and Qwen2.5-VL-7B reaches 76.4%. But MIDA contains a large number of neutral utterances, so a model can post a decent-looking score simply by choosing NEUTRAL often.

Once the researchers keep only the samples that must be judged true or false, the picture changes sharply. GPT-4o records binary accuracy of just 27.6%, while GPT-4o-mini reaches 39.4%. Qwen2.5-VL-7B has the highest overall accuracy, yet its F1 score on FALSE is 0.

University of Tokyo’s MIDA benchmark finds top multimodal models still struggle to detect deception in group play 5

In other words, a model may answer “cannot determine” most of the time, earn a relatively high overall score, and still fail to identify lies in any meaningful sense.

The weakest results appear in the categories that require the most social reasoning. For identity-claim utterances, the average accuracy across models is only 15.7% on one subset and 25.3% on the other. For evidence-based utterances, average accuracy stays below 40% in both cases. A model may recognize that a statement is making an accusation, but that does not mean it can tell whether the accusation is actually possible.

Experienced players make the task harder still. In the YouTube subset, the share of false evidence reaches 36.8%, close to three times the level seen among beginners. The share of identity lies is 51.6%, well above the 18.9% reported for novice players.

Dialogue history matters more than extra visual detail

The team ran ablation studies to measure the contribution of different inputs. When prior dialogue history is removed, GPT-4o-mini’s accuracy on true-or-false samples falls from 39.4% to 13.4%.

University of Tokyo’s MIDA benchmark finds top multimodal models still struggle to detect deception in group play 6

That result suggests a split between local and social understanding. A model may only need local text to tell whether a sentence is a question, a defense or an accusation. But to judge whether it is true, the model has to remember who said what, who knows what, and whether different players’ statements support or contradict one another.

More visual information does not automatically help. Adding one frame raises macro F1 by 2.5 percentage points. Expanding that to three frames reduces the gain to 2.1 points. More frames bring more facial expressions and body movements, but they also add more noise. The model may see more cues without knowing which ones actually matter.

SoCoT and DSEM take different approaches

To address these issues, the researchers designed two modules: Social Chain-of-Thought, or SoCoT, and Dynamic Social Episodic Memory, or DSEM.

SoCoT asks the model to first describe facial expression, body posture and vocal traits, then infer the speaker’s goal and mental state, and only after that make a truthfulness judgment. The idea is to break an intuition such as “the speaker’s voice is shaking, so something may be off” into reasoning steps that can be inspected.

University of Tokyo’s MIDA benchmark finds top multimodal models still struggle to detect deception in group play 7

But nonverbal cues do not always improve judgment. In GPT-4o-mini, analyzing facial cues alone raises overall accuracy by 8 percentage points, yet binary accuracy on true-versus-false statements drops by 7.1 points. The paper argues that behaviors that look suspicious, such as nervousness or avoiding eye contact, are not the same thing as lying. A model may detect those behaviors and still interpret them incorrectly.

DSEM works differently. It maintains a continuously updated memory board for each player, recording role claims, actions, observations, known facts, interpersonal relations and the event timeline. When a new utterance appears, the model does not focus only on the current frame. It checks what that person should know at that point, whether the statement conflicts with earlier information, and whether other players’ remarks support it.

In the experiments, DSEM produces more stable gains. It improves GPT-4o-mini’s binary accuracy by 2.3 percentage points and macro F1 by 3.3 points. On Qwen2.5-VL-7B, it lifts macro precision by 7.6 points. Compared with searching for a tell in a single moment, tracking each participant’s knowledge and stance over time appears closer to how humans actually reason in social settings.

What the paper says this means

The authors argue that MIDA is not only about Werewolf. In meeting collaboration, online discussion, human-computer interaction, intelligent customer service and multi-agent collaboration, models also need to track what different participants know, how their positions change and how they affect one another.

University of Tokyo’s MIDA benchmark finds top multimodal models still struggle to detect deception in group play 8

In the paper’s framing, current multimodal large models are powerful “knowledge engines,” but not yet mature “social agents.” What they lack may not simply be a stronger visual encoder. The missing piece may be the ability to continuously model other people’s knowledge, beliefs and intentions, which is commonly described as theory of mind. Once a model leaves visual question answering with clear answers and enters an environment where participants hold different private information, views keep shifting and people actively try to influence one another, even today’s strongest multimodal systems still have trouble truly reading the room.

Public materials include the paper at https://arxiv.org/abs/2511.16221, the project page at https://ut-vision-social.github.io/, the code repository at https://github.com/kkkcx/Can-MLLMs-Read-the-Room and the dataset at https://huggingface.co/datasets/Ukpkmkkk/MIDA.

The article credits the WeChat public account Xinzhiyuan, with authorship listed as Xinzhiyuan.

This article was originally published by Bit.Fan. For more cryptocurrency news and market insights, visit www.bit.fan.
6600

Disclaimer:

The market information, project data, and third-party content displayed on this platform are for industry information sharing only and do not constitute any form of investment advice or return commitment.

Cryptocurrency trading carries high risks. Users should fully assess their risk tolerance and make independent decisions. All profits, losses, and legal responsibilities are borne by the users themselves.

University of Tokyo’s MIDA benchmark finds top multimodal models still struggle to detect deception in group play | Bit.Fan