Fudan University’s CVL Lab has released AVTrack, a benchmark for audio-visual speaker tracking in complex human-centered scenes. The paper has been accepted to ICML 2026, and the project page, paper, code, and dataset are now public.
Audio-visual speaker tracking aims to let a model use both sound and video to locate and continuously follow the person who is speaking. The same core capability sits behind use cases such as automatically focusing on the active speaker in a multi-person online meeting, identifying who is talking in entertainment footage for subtitle generation, and tracking a speaking person through surveillance video.
AVTrack is designed for more than simple speaker detection
The task is straightforward in wording but difficult in practice. Given a video with audio, a model must not only decide who is speaking at a given moment, but also keep that person’s identity stable as the video changes and as speech turns move across different segments.
That becomes much harder in real footage. People can block one another, move frequently, and alternate turns. Camera views may switch. Some clips also contain off-screen narration or other cases where the visible person and the audio source do not line up cleanly.
AVTrack targets this kind of setting. It requires a model to jointly interpret sound and visuals, generate a pixel-level mask for each active speaker, and preserve a stable instance identity over time. In practical terms, the system has to answer three questions at once: who is speaking, where that person is in each frame, and whether the person seen now is still the same instance across occlusion, camera switches, or position swaps.
The third part is central. A model may find a speaker in one frame by relying on appearance or mouth motion, but maintaining the right identity through a long video calls for handling occlusion, alternating speakers, and mismatches between what is heard and what is visible.
Earlier benchmarks leave gaps in dynamic and crowded scenes
According to the paper, many early datasets for audio-visual speaker tracking came from controlled lab environments with few people and relatively fixed camera and microphone positions. In those settings, a model could post strong results without really understanding temporal dynamics or cross-modal reasoning. Static co-occurrence between sound and image was often enough.
Audio-Visual Segmentation (AVS) and Audio-Visual Instance Segmentation (AVIS) tasks have drawn broad attention in recent years, and progress has been substantial. Even so, the team argues that one basic question remains under-tested: when a scene becomes dynamic, crowded, and discontinuous, how well can current models preserve audio-visual correspondence and long-range identity association?
Later audio-visual segmentation benchmarks introduced pixel-level annotation and more natural video sources, but many clips still run only 5 to 10 seconds. That makes them less suitable for testing long-term temporal modeling. AVISeg extended average video length to around 60 seconds and added instance-level segmentation and tracking labels, yet coverage of camera motion, occlusion, background changes, and relative position shifts remains limited.
This creates an evaluation blind spot. A model may perform well in relatively static footage while still leaning on short-term co-occurrence between sound and image. Once a person disappears and reappears, the shot changes, or the most visually salient person is not the one producing the audio, existing benchmarks do not fully reveal whether the model has kept the cross-modal link intact.
871 videos and eight diagnostic challenge types
AVTrack contains 871 videos with an average duration of 54.0 seconds. It provides 3,120 pixel-level instance trajectories with cross-frame identities. The data spans six source categories: dramas, vlogs, animation, reality shows, interviews, and stage performances.
Unlike a standard training set, AVTrack is built as a pure test benchmark. The idea is to use it as an independent test set and examine whether models can transfer to difficult human-centered scenes.
The benchmark is organized around eight diagnostic challenge types:
- Visual Occlusion: the speaker is partially blocked by other people or objects, creating overlap and blurred boundaries. This property appears in 80.9% of AVTrack samples, versus 8.6% in AVISeg.
- Relative Position Change: the spatial relationship between instances changes over time, such as a speaker moving from one side of another person to the other. AVTrack covers this in 70.7% of samples, compared with 9.7% in AVISeg.
- Background Switch: a notable change in scene or background, such as moving from indoors to outdoors, weakens continuity based on background cues. This accounts for 60.5% of AVTrack and 5.9% of AVISeg.
- Camera Motion Change: camera movement, zoom, panning, viewpoint changes, or shot switches alter scale, angle, and visible area. This is the most common challenge in AVTrack, covering 90.5% of samples, versus 7.1% in AVISeg.
- Multiple Instances: several candidate people appear in the frame, but only some are speaking, so the model must link the audio to the right visual instance. This appears in 64.9% of AVTrack and 13.6% of AVISeg.
- Multi-turn Sounding: different people speak in alternating turns over time, forcing the model to update target identity dynamically. This covers 56.8% of AVTrack and 16.0% of AVISeg.
- Audio-Visual Inconsistency: the audio does not map directly to any currently visible person, as in voice-over, off-screen speech, or a speaker who is temporarily out of frame. About 9.2% of AVTrack samples contain this condition.
- Instance Scale Dynamics: the target instance shows large scale differences or large scale changes over time, raising the difficulty of small-object perception and cross-scale tracking. This appears in 56.9% of AVTrack and 35.3% of AVISeg.
The paper says these properties do two jobs at once: they describe scene complexity and help diagnose specific failure modes. On dynamic visual changes, AVTrack is much tougher than AVISeg. Camera motion changes appear in 90.5% of videos, visual occlusion in 80.9%, and relative position change in 70.7%, far above AVISeg’s 7.1%, 8.6%, and 9.7%.
To build the dataset, annotators first screened candidate clips from roughly 1,300 public videos. They used Grounded-SAM for automatic pre-annotation, then corrected masks frame by frame and checked temporal identities manually. The data collection, annotation, and validation process lasted nearly three months and involved 15 professional annotators.
Current methods still struggle on AVTrack
The team evaluated representative Video Instance Segmentation (VIS) and audio-visual instance segmentation methods under a unified setup. VIS models were pre-trained on YouTube-VIS and then fine-tuned on AVISeg. AVIS methods were trained on AVISeg. All of them were then tested directly on AVTrack.
The results were modest. Visual-only methods VITA, LBVQ, and CAVIS all scored below 12 on the HOTA test metric. Adding audio improved the picture: AVISM reached 20.84 and ACVIS reached 20.60. That suggests audio supplies useful information, but long-term identity association remains difficult.
The researchers also provided two complementary baselines. AVTrackFormer, an end-to-end model, posted a HOTA score of 21.47. AVTracker, a training-free modular model, reached 29.08.
Relative to AVISM, the strongest existing AVIS method in the comparison, AVTracker improved by 8.24 points, from 20.84 to 29.08. Relative to AVTrackFormer, it improved by 7.61 points, from 21.47 to 29.08.
The team also tested Gemini 2.5 Pro in a zero-shot setting. After feeding it video frames at 1 FPS together with raw audio, they reported a HOTA score of 14.4. The paper uses that result to argue that broad audio-visual understanding in a general omni-modal large language model does not directly become stable pixel-level speaker tracking in hard scenes.
AVTracker breaks long-range association into three stages
AVTracker’s main idea is to split the long-video association problem into three steps.
First, the system uses Whisper to obtain timestamped speech segments. It then uses EC APA-TDNN speaker embedding to merge adjacent segments belonging to the same speaker, producing more compact speaker segments.
Next, within each local time window, SAM3 supplies candidate person masks. A Local Reasoner combines speech transcription and visual observations to match the current speech segment to a visible person and form short local tracks.
Finally, a Global Reasoner gathers key frames from local tracks and performs identity grouping across the full video, restoring continuous speaker trajectories over time.
Ablation results support the local-to-global design
Ablation experiments backed up that design choice. Under the same model setting, removing local segment compression dropped HOTA from 24.01 to 16.88. Turning off the dynamic window mechanism in the full model lowered HOTA from 28.85 to 27.45.
Speech separation did not always help. Using MossFormer2 raised HOTA from 28.85 to 29.08, but using SepFormer reduced it to 28.41. That suggests errors introduced during speech separation can propagate into later audio-visual alignment.
Audio-visual inconsistency remains the hardest case
Across fine-grained categories, audio-visual inconsistency was still the toughest setting for every method tested. Even AVTracker reached only 18.5 HOTA there.
The paper also notes failure cases in visually occluded and multi-instance scenes. When nearby speakers are partially blocked and alternate frequently, the global reasoner may still merge different identities by mistake or split one identity into multiple tracks.
Those failure modes point to open research problems in long-term memory, cross-window evidence accumulation, and identity reasoning that can be corrected after initial errors. The team said future work will continue to study hard settings such as audio-visual inconsistency and instance scale dynamics, while also exploring memory-enhanced tracking, reflection-based error correction, and more efficient data and model design.
Project links are available
With AVTrack now formally released, the researchers say the benchmark is intended to help the community examine a basic question more precisely: whether a model is merely finding the person who looks like a speaker in a static frame, or whether it has learned to keep track of who is speaking, when, and where in a changing scene.
The public links provided in the source are:
- Project page: https://fudancvl.github.io/AVTrack/
- Paper: https://arxiv.org/abs/2606.02724
- Code: https://github.com/FudanCVL/AVTrack
- Dataset: https://huggingface.co/datasets/FudanCVL/AVTrack
The source article states that the piece originally came from the WeChat public account "新智元," authored by LRST.