# LingoQA ![Category](https://img.shields.io/badge/Category-Driving%20Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Lingo--Judge%20Score-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-1%2C000%2B-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Medium-d29922) ![Created](https://img.shields.io/badge/Created-2024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2312.14115) [![GitHub](https://img.shields.io/badge/GitHub-wayveai%2FLingoQA-181717?logo=github&logoColor=white)](https://github.com/wayveai/LingoQA) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/wayveai/LingoQA) > **Quick answer:** LingoQA is a video question-answering benchmark created by Wayve that tests whether a model can reason about driving scenarios — explaining actions, hazards, and scene context from short driving video clips. Models are scored with the Lingo-Judge metric, a learned evaluator trained to approximate human judgment of free-form driving-reasoning answers. As of August 2026, NVIDIA's Alpamayo 2 Super leads the public leaderboard at 79.2 among nearly 40 evaluated models. ## At a Glance **What it tests:** Free-form question answering over short driving video clips, covering scene description, action justification, and hazard/behavior reasoning. **Why it matters:** Autonomous driving systems increasingly need to explain their decisions, not just execute them. LingoQA is one of the few standardized benchmarks measuring whether a model's driving-scenario reasoning is coherent and correct, not just its trajectory accuracy. **Known limitations:** The Lingo-Judge metric is itself a trained model, so scores can be sensitive to judge calibration; general-purpose VLMs (GPT-4o, Gemini 2.5 Pro) are evaluated zero-shot alongside AV-specialist models fine-tuned specifically for the task, which is not always an apples-to-apples comparison. ## What LingoQA Measures LingoQA presents a model with a short multi-camera driving video clip and a natural-language question about the scene — what the vehicle is doing, why, what hazards are present, or what should happen next. The model must produce a free-form text answer grounded in the visual content, testing scene understanding and causal reasoning rather than simple object detection. Because driving reasoning has no single "correct" phrasing, LingoQA does not grade with exact-match or multiple-choice scoring. Instead, it uses Lingo-Judge, a purpose-trained evaluator model that scores free-form answers against reference explanations, approximating how a human safety reviewer would judge whether the reasoning is sound. A high Lingo-Judge score means a model reliably identifies the right causal factors behind a driving decision, not just the right trajectory. The public leaderboard spans three categories of systems: AV/embodied specialists fine-tuned specifically for driving reasoning (e.g., Alpamayo, RoboTron-Drive, ACE-Brain-0), general-purpose open-source VLMs evaluated zero-shot or fine-tuned (Qwen3-VL, Qwen2.5-VL, InternVL), and closed-source proprietary models (GPT-4V, GPT-4o, Gemini 2.5 Pro). This mix makes LingoQA one of the few benchmarks that directly compares specialist AV reasoning models against general frontier VLMs on the same driving-scenario tasks. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning (autonomous driving, video QA) | | Metric | Lingo-Judge score (0–100) | | Number of tasks | 1,000+ evaluation QA pairs | | Saturation | Medium | | Created by | Marcu et al. (Wayve) | | Source paper | [Marcu et al. 2024](https://arxiv.org/abs/2312.14115) | | GitHub | [wayveai/LingoQA](https://github.com/wayveai/LingoQA) | | Dataset | [wayveai/LingoQA](https://huggingface.co/datasets/wayveai/LingoQA) | ## How LingoQA Is Scored Each model answer is compared against reference explanations using Lingo-Judge, a lightweight evaluator model fine-tuned specifically to approximate human ratings of driving-reasoning quality. The score is reported on a 0–100 scale, where scores in the 70s and above indicate the model consistently identifies correct causal factors (hazards, right-of-way, why an action was taken), while scores below 50 indicate frequent reasoning errors or generic, ungrounded answers. Because Lingo-Judge is itself learned rather than rule-based, absolute scores can shift slightly between judge versions — cross-paper comparisons are most reliable when the same judge checkpoint is used, as is the case for the leaderboard below. ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Alpamayo 2 Super](/models/nvidia/alpamayo-2-super) | 79.2 | [NVIDIA model card](https://huggingface.co/nvidia/Alpamayo2-Super) | 2026-08 | | 3 | [Qwen3-VL 32B Instruct](/models/qwen/qwen3-vl-32b-instruct) | 72.2 | [NVIDIA model card](https://huggingface.co/nvidia/Alpamayo2-Super) | 2026-08 | | 7 | [Gemini 2.5 Pro](/models/google/gemini-2-5-pro) | 64.1 | [NVIDIA model card](https://huggingface.co/nvidia/Alpamayo2-Super) | 2026-08 | | 14 | [Qwen3-VL 8B Instruct](/models/qwen/qwen3-vl-8b-instruct) | 57.0 | [NVIDIA model card](https://huggingface.co/nvidia/Alpamayo2-Super) | 2026-08 | | 18 | [GPT-4o](/models/openai/gpt-4o) | 56.0 | [NVIDIA model card](https://huggingface.co/nvidia/Alpamayo2-Super) | 2026-08 | *Scores sourced from NVIDIA's published Alpamayo 2 Super model card and technical blog (August 4, 2026), which re-evaluated general-purpose VLMs against the same Lingo-Judge checkpoint alongside AV-specialist models. Rank numbers reflect position in NVIDIA's full ~40-model comparison chart; only models with existing Benchgen pages are listed here. Results depend on judge version and evaluation harness — see the source for full methodology.* ## LingoQA on Benchgen > No Benchgen results yet — [be the first to run LingoQA](/benchmarks/lingoqa). ## LingoQA vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **LingoQA** | Driving-scenario video QA, causal reasoning | 1,000+ | Medium | | [Video-MME](/benchmarks/video-mme) | General video understanding across domains | 900 | Medium | | [MMVU](/benchmarks/mmvu) | Multi-discipline video reasoning | N/A | Medium | | [MMMU-Pro](/benchmarks/mmmu-pro) | Multimodal college-level reasoning (static images) | N/A | High | LingoQA is domain-specific to driving scenarios, unlike general video-reasoning benchmarks like Video-MME or MMVU. Teams evaluating models for AV or robotics reasoning should prioritize LingoQA over general multimodal benchmarks, which don't test causal, safety-relevant driving judgment. ## Run LingoQA on Your Model BenchGen lets teams track LingoQA-style driving-reasoning performance across model versions and harness changes — turning a one-time vendor-reported score into a continuous regression signal as your AV stack's underlying model or prompting changes. [Start tracking on BenchGen](/benchmarks/lingoqa). ## Related Models & Benchmarks - Top performers: [Alpamayo 2 Super](/models/nvidia/alpamayo-2-super) · [Qwen3-VL 32B Instruct](/models/qwen/qwen3-vl-32b-instruct) · [Gemini 2.5 Pro](/models/google/gemini-2-5-pro) - Related benchmarks: [Video-MME](/benchmarks/video-mme) · [MMVU](/benchmarks/mmvu) · [MMMU-Pro](/benchmarks/mmmu-pro) ## Frequently Asked Questions

What is LingoQA?

LingoQA is a video question-answering benchmark created by Wayve that tests a model's ability to reason about driving scenarios — explaining actions, hazards, and scene context — scored using the Lingo-Judge metric.

What does a good LingoQA score look like?

Scores in the 70s and above (out of 100) indicate a model reliably identifies correct causal factors behind driving decisions. As of August 2026, the top score is 79.2 (Alpamayo 2 Super); general-purpose frontier VLMs like GPT-4o and Gemini 2.5 Pro score in the mid-50s to low-60s range.

Who created LingoQA?

LingoQA was created by Marcu et al. at Wayve, an autonomous driving company, and published as an open benchmark with code and dataset available on GitHub and HuggingFace.

Is LingoQA saturated?

Saturation is medium. Top AV-specialist models now score in the high 70s, but general-purpose frontier VLMs still trail meaningfully (mid-50s to mid-60s), leaving room for improvement, particularly for models not fine-tuned on driving-specific data.

How does LingoQA differ from other video reasoning benchmarks?

Unlike general video benchmarks such as Video-MME or MMVU, LingoQA is narrowly focused on driving scenarios and causal reasoning about vehicle behavior, making it a domain-specific complement rather than a substitute for general multimodal video benchmarks.

--- *Benchmark definition paraphrased from [Marcu et al. 2024](https://arxiv.org/abs/2312.14115). State-of-the-art scores sourced from NVIDIA's published Alpamayo 2 Super model card and blog (August 4, 2026) and attributed inline. Last updated 2026-08-04.*

LingoQA — Results

LingoQA — Evaluations