Benchgen

MMAR — Results

RankModelScore
1lfm3-omni-2b51.4
M

MMAR

1 phaseActive

A 1,000-item benchmark for deep reasoning across speech, audio, music, and mixed-modality audio, with four hierarchical reasoning layers (Signal, Perception, Semantic, Cultural).

Overview

MMAR

Category Metric Saturation Created

Paper

Quick answer: MMAR ("A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix") is a benchmark introduced by Ziyang Ma, Yinghao Ma, Yanqiao Zhu, et al. (Shanghai Jiao Tong University and collaborators, submitted May 2025) with 1,000 curated audio-question-answer triplets drawn from real-world internet videos. Unlike benchmarks limited to one domain of sound, MMAR spans mixed-modality combinations of sound, music, and speech, and grades models across four hierarchical reasoning layers: Signal, Perception, Semantic, and Cultural.

What MMAR Measures

Most audio benchmarks test a single domain (speech-only, music-only, or environmental-sound-only). MMAR instead curates real-world audio that mixes these domains together and requires multi-step reasoning beyond surface-level recognition — each question ships with a Chain-of-Thought rationale, and a subset demands graduate-level domain knowledge, raising the difficulty bar versus simpler audio QA sets.

Public Benchmark Scores

ModelScoreSourceDate
LFM3-Omni-2B51.40%Liquid AI model card2026-10-27
  • LFM3-Omni-2B — currently the only Benchgen-tracked model with a public MMAR score.

Sources

  • Ma, Ziyang, Ma, Yinghao, Zhu, Yanqiao, et al. "MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix." arXiv:2505.13032, submitted 2025-05-19. Open-source at https://github.com/ddlBoJack/MMAR.