LingoQA

LingoQA

Quick answer: LingoQA is a video question-answering benchmark created by Wayve that tests whether a model can reason about driving scenarios — explaining actions, hazards, and scene context from short driving video clips. Models are scored with the Lingo-Judge metric, a learned evaluator trained to approximate human judgment of free-form driving-reasoning answers. As of August 2026, NVIDIA's Alpamayo 2 Super leads the public leaderboard at 79.2 among nearly 40 evaluated models.

At a Glance

What it tests: Free-form question answering over short driving video clips, covering scene description, action justification, and hazard/behavior reasoning.

Why it matters: Autonomous driving systems increasingly need to explain their decisions, not just execute them. LingoQA is one of the few standardized benchmarks measuring whether a model's driving-scenario reasoning is coherent and correct, not just its trajectory accuracy.

Known limitations: The Lingo-Judge metric is itself a trained model, so scores can be sensitive to judge calibration; general-purpose VLMs (GPT-4o, Gemini 2.5 Pro) are evaluated zero-shot alongside AV-specialist models fine-tuned specifically for the task, which is not always an apples-to-apples comparison.

What LingoQA Measures

LingoQA presents a model with a short multi-camera driving video clip and a natural-language question about the scene — what the vehicle is doing, why, what hazards are present, or what should happen next. The model must produce a free-form text answer grounded in the visual content, testing scene understanding and causal reasoning rather than simple object detection.

Because driving reasoning has no single "correct" phrasing, LingoQA does not grade with exact-match or multiple-choice scoring. Instead, it uses Lingo-Judge, a purpose-trained evaluator model that scores free-form answers against reference explanations, approximating how a human safety reviewer would judge whether the reasoning is sound. A high Lingo-Judge score means a model reliably identifies the right causal factors behind a driving decision, not just the right trajectory.

The public leaderboard spans three categories of systems: AV/embodied specialists fine-tuned specifically for driving reasoning (e.g., Alpamayo, RoboTron-Drive, ACE-Brain-0), general-purpose open-source VLMs evaluated zero-shot or fine-tuned (Qwen3-VL, Qwen2.5-VL, InternVL), and closed-source proprietary models (GPT-4V, GPT-4o, Gemini 2.5 Pro). This mix makes LingoQA one of the few benchmarks that directly compares specialist AV reasoning models against general frontier VLMs on the same driving-scenario tasks.

Benchmark Specifications

How LingoQA Is Scored

Each model answer is compared against reference explanations using Lingo-Judge, a lightweight evaluator model fine-tuned specifically to approximate human ratings of driving-reasoning quality. The score is reported on a 0–100 scale, where scores in the 70s and above indicate the model consistently identifies correct causal factors (hazards, right-of-way, why an action was taken), while scores below 50 indicate frequent reasoning errors or generic, ungrounded answers. Because Lingo-Judge is itself learned rather than rule-based, absolute scores can shift slightly between judge versions — cross-paper comparisons are most reliable when the same judge checkpoint is used, as is the case for the leaderboard below.

State-of-the-Art Results

Scores sourced from NVIDIA's published Alpamayo 2 Super model card and technical blog (August 4, 2026), which re-evaluated general-purpose VLMs against the same Lingo-Judge checkpoint alongside AV-specialist models. Rank numbers reflect position in NVIDIA's full ~40-model comparison chart; only models with existing Benchgen pages are listed here. Results depend on judge version and evaluation harness — see the source for full methodology.

LingoQA on Benchgen

No Benchgen results yet — be the first to run LingoQA.

LingoQA vs Other Benchmarks

LingoQA is domain-specific to driving scenarios, unlike general video-reasoning benchmarks like Video-MME or MMVU. Teams evaluating models for AV or robotics reasoning should prioritize LingoQA over general multimodal benchmarks, which don't test causal, safety-relevant driving judgment.

Run LingoQA on Your Model

BenchGen lets teams track LingoQA-style driving-reasoning performance across model versions and harness changes — turning a one-time vendor-reported score into a continuous regression signal as your AV stack's underlying model or prompting changes. Start tracking on BenchGen.

Related Models & Benchmarks

• Top performers: Alpamayo 2 Super · Qwen3-VL 32B Instruct · Gemini 2.5 Pro
• Related benchmarks: Video-MME · MMVU · MMMU-Pro

Frequently Asked Questions

Benchmark definition paraphrased from Marcu et al. 2024. State-of-the-art scores sourced from NVIDIA's published Alpamayo 2 Super model card and blog (August 4, 2026) and attributed inline. Last updated 2026-08-04.