ChartQAPro

ChartQAPro

Quick answer: ChartQAPro is a more diverse and challenging chart question answering benchmark built to restore difficulty lost to saturation on ChartQA — it spans 1,341 charts from 157 sources (including infographics and dashboards) with 1,948 questions across multiple-choice, conversational, hypothetical, and unanswerable formats, causing sharp score drops for models that appeared near-saturated on the original ChartQA.

At a Glance

What it tests: Real-world chart reasoning across diverse chart types and question formats, including questions designed to be intentionally unanswerable from the given chart — testing whether models can recognize the limits of available information rather than always guessing an answer.

Why it matters: ChartQA, once a standard chart-understanding benchmark, has become saturated by modern LVLMs; ChartQAPro's broader diversity in chart sources and question types exposes a dramatic performance gap that ChartQA alone was hiding, e.g., Claude Sonnet 3.5 scores 90.5% on ChartQA but only 55.81% on ChartQAPro.

Known limitations: As a relatively new benchmark (2025), less longitudinal model coverage exists compared to more established chart benchmarks, and question-type-specific scoring (multiple-choice, conversational, hypothetical, unanswerable) requires careful per-type analysis rather than a single blended number.

What ChartQAPro Measures

ChartQAPro includes 1,341 charts from 157 diverse real-world sources, spanning many chart types including infographics and dashboards that go well beyond the simpler bar/line/pie charts common in earlier chart QA datasets. The benchmark features 1,948 questions across varied formats — multiple-choice, conversational (multi-turn), hypothetical ("what if" reasoning), and deliberately unanswerable questions — designed to better reflect the real-world diversity and ambiguity that chart-reading users actually encounter.

The paper's evaluation of 21 models revealed a substantial performance drop moving from ChartQA to ChartQAPro; for example, Claude Sonnet 3.5 dropped from 90.5% on ChartQA to just 55.81% on ChartQAPro, underscoring how much genuine chart-reasoning complexity the original benchmark was failing to capture. Detailed error analyses and ablation studies in the paper identify specific challenges — like handling unanswerable questions gracefully and correctly parsing dense infographic layouts — as key opportunities for advancing LVLM chart understanding.

ChartQAPro is supported in the OpenCompass VLMEvalKit toolkit for standardized, reproducible evaluation.

Benchmark Specifications

How ChartQAPro Is Scored

Each question type is scored with metrics appropriate to its format — typically ANLS (Average Normalized Levenshtein Similarity) or accuracy — then reported per question type and as an overall composite score, using the official evaluation script that follows the methodology from the paper.

State-of-the-Art Results

Scores vary substantially by question type; models that appear strong on ChartQA can score far lower on ChartQAPro. See the paper and VLMEvalKit leaderboard for current standings.

ChartQAPro on Benchgen

No Benchgen results yet — be the first to run ChartQAPro.

ChartQAPro vs Other Benchmarks

ChartQAPro is the harder, more diverse successor to ChartQA, purpose-built to remain unsaturated as frontier models continue improving on chart-specific benchmarks.

Run ChartQAPro on Your Model

Benchgen lets teams run ChartQAPro against their own model or agent versions on demand, tracking scores broken out by question type (including unanswerable questions) to catch regressions in real-world chart reasoning.

Related Models & Benchmarks

Explore related multimodal benchmarks like MME-RealWorld and MMBench-Video to compare chart-specific reasoning with broader real-world perception and video understanding evaluation.

Frequently Asked Questions