Benchgen

AV-SpeakerBench — Results

RankModelScore
1lfm3-omni-2b46.79
A

AV-SpeakerBench

1 phaseActive

A 3,212-question multiple-choice benchmark for speaker-centric audiovisual reasoning in real-world videos, with fusion-grounded question design that embeds audiovisual dependencies into question semantics.

Overview

AV-SpeakerBench

Category Metric Saturation Created

Paper

Quick answer: AV-SpeakerBench ("See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models") is a benchmark introduced by Le Thien Phuc Nguyen, Zhuoran Yu, Samuel Low Yu Hang, Subin An, Jeongik Lee, Yohan Ban, SeungEun Chung, Thanh-Huy Nguyen, JuWan Maeng, Soochahn Lee, and Yong Jae Lee (submitted December 2025, accepted as Findings of CVPR 2026), featuring 3,212 multiple-choice questions focused on speaker-centric audiovisual reasoning in real-world videos.

What AV-SpeakerBench Measures

Most video benchmarks remain visually solvable or only coarsely evaluate speech, offering limited insight into whether models can align who speaks, what is said, and when it occurs. AV-SpeakerBench treats speakers — not scenes — as the core reasoning unit, with fusion-grounded question design embedding audiovisual dependencies directly into question semantics, and expert-curated annotations ensuring temporal precision and cross-modal validity. The paper's own evaluations found the Gemini family consistently outperforming open-source systems, with Gemini 2.5 Pro achieving the best results, while among open models Qwen3-Omni-30B approaches Gemini 2.0 Flash but remains behind Gemini 2.5 Pro — primarily due to weaker audiovisual fusion rather than visual perception.

Public Benchmark Scores

ModelScoreSourceDate
LFM3-Omni-2B46.79%Liquid AI model card2026-10-27
  • LFM3-Omni-2B — currently the only Benchgen-tracked model with a public AV-SpeakerBench score.
  • WorldSense — a related real-world omnimodal benchmark also scored by LFM3-Omni-2B.
  • Daily-Omni — a related audio-visual temporal-alignment benchmark also scored by LFM3-Omni-2B.

Sources

  • Nguyen, Le Thien Phuc, Yu, Zhuoran, Hang, Samuel Low Yu, An, Subin, Lee, Jeongik, Ban, Yohan, Chung, SeungEun, Nguyen, Thanh-Huy, Maeng, JuWan, Lee, Soochahn, Lee, Yong Jae. "See, Hear, and Understand: Benchmarking Audiovisual Human Speech Understanding in Multimodal Large Language Models." arXiv:2512.02231, submitted 2025-12-01, revised 2026-04-10 (v2), Findings of CVPR 2026.