VoiceBench

VoiceBench

Quick answer: VoiceBench is the first comprehensive benchmark for LLM-based voice assistants (Chen et al., TACL 2026, National University of Singapore). It evaluates models on diverse spoken instructions including both real human voices and synthetic speech, covering open-ended QA, multiple-choice, instruction following, reasoning, and safety tasks. The composite score aggregates rule-based and GPT-judge metrics across multiple subsets. Note: VoiceBench uses rule-based string matching for some evaluations, making scores sensitive to output formatting. Gemini 3.1 Pro leads the Inkling comparison at 94.3%.

At a Glance

What it tests: An end-to-end voice assistant's ability to correctly understand and respond to spoken instructions varying in speaker characteristics (accent, age, noise), content types (factual, reasoning, creative), and difficulty (single-turn, multi-turn, safety).

Why it matters: Most voice assistant evaluations focus on ASR quality or general knowledge with clean speech, missing the real-world complexity of voice interactions. VoiceBench introduces multiple speech variation factors — diverse speaker characteristics, environmental noise, and challenging linguistic content — providing a more realistic evaluation of voice AI.

Known limitations: Some VoiceBench subsets use rule-based, hard-coded string matching for grading. The Inkling model card explicitly notes this "makes the evaluation sensitive to output-formatting differences" — models that produce correct answers in different formats may score lower than models matching expected strings exactly.

Benchmark Specifications

State-of-the-Art Results

Scores from Inkling model card (Thinking Machines Lab, July 2026). Only 2 models reported (native audio input required; other models not evaluated on this benchmark).

Last updated 2026-07-16.