# ChartQAPro ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-ANLS%20%2F%20Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-1%2C948-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Apr%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2504.05506) [![GitHub](https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white)](https://github.com/vis-nlp/ChartQAPro) > **Quick answer:** ChartQAPro is a more diverse and challenging chart question answering benchmark built to restore difficulty lost to saturation on ChartQA — it spans 1,341 charts from 157 sources (including infographics and dashboards) with 1,948 questions across multiple-choice, conversational, hypothetical, and unanswerable formats, causing sharp score drops for models that appeared near-saturated on the original ChartQA. ## At a Glance **What it tests:** Real-world chart reasoning across diverse chart types and question formats, including questions designed to be intentionally unanswerable from the given chart — testing whether models can recognize the limits of available information rather than always guessing an answer. **Why it matters:** ChartQA, once a standard chart-understanding benchmark, has become saturated by modern LVLMs; ChartQAPro's broader diversity in chart sources and question types exposes a dramatic performance gap that ChartQA alone was hiding, e.g., Claude Sonnet 3.5 scores 90.5% on ChartQA but only 55.81% on ChartQAPro. **Known limitations:** As a relatively new benchmark (2025), less longitudinal model coverage exists compared to more established chart benchmarks, and question-type-specific scoring (multiple-choice, conversational, hypothetical, unanswerable) requires careful per-type analysis rather than a single blended number. ## What ChartQAPro Measures ChartQAPro includes 1,341 charts from 157 diverse real-world sources, spanning many chart types including infographics and dashboards that go well beyond the simpler bar/line/pie charts common in earlier chart QA datasets. The benchmark features 1,948 questions across varied formats — multiple-choice, conversational (multi-turn), hypothetical ("what if" reasoning), and deliberately unanswerable questions — designed to better reflect the real-world diversity and ambiguity that chart-reading users actually encounter. The paper's evaluation of 21 models revealed a substantial performance drop moving from ChartQA to ChartQAPro; for example, Claude Sonnet 3.5 dropped from 90.5% on ChartQA to just 55.81% on ChartQAPro, underscoring how much genuine chart-reasoning complexity the original benchmark was failing to capture. Detailed error analyses and ablation studies in the paper identify specific challenges — like handling unanswerable questions gracefully and correctly parsing dense infographic layouts — as key opportunities for advancing LVLM chart understanding. ChartQAPro is supported in the OpenCompass VLMEvalKit toolkit for standardized, reproducible evaluation. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (chart question answering) | | Metric | ANLS/accuracy (%), reported per question type (multiple-choice, conversational, hypothetical, unanswerable) | | Number of tasks | 1,948 questions across 1,341 charts from 157 sources | | Saturation | Low — sharp score drops versus ChartQA reveal substantial remaining headroom | | Created by | Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, and collaborators | | Source paper | [Masry et al. 2025](https://arxiv.org/abs/2504.05506) | | Code | [GitHub](https://github.com/vis-nlp/ChartQAPro) | | Dataset | [Hugging Face](https://huggingface.co/datasets/ahmed-masry/ChartQAPro) | ## How ChartQAPro Is Scored Each question type is scored with metrics appropriate to its format — typically ANLS (Average Normalized Levenshtein Similarity) or accuracy — then reported per question type and as an overall composite score, using the official evaluation script that follows the methodology from the paper. ## State-of-the-Art Results > Scores vary substantially by question type; models that appear strong on ChartQA can score far lower on ChartQAPro. See the [paper](https://arxiv.org/abs/2504.05506) and [VLMEvalKit](https://github.com/open-compass/VLMEvalKit) leaderboard for current standings. ## ChartQAPro on Benchgen > No Benchgen results yet — [be the first to run ChartQAPro](/benchmarks/masry-et-al-york-university/chartqapro). ## ChartQAPro vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **ChartQAPro** | Diverse, challenging chart QA with unanswerable questions | 1,948 | low | | [MME-RealWorld](/benchmarks/zhang-et-al-xiamen-university/mme-realworld) | High-resolution real-world perception across 5 scenarios | 29,429 | low | | [MMBench-Video](/benchmarks/fang-et-al-shanghai-ai-laboratory/mmbench-video) | Long-form, multi-shot video understanding | 1,998 | low | ChartQAPro is the harder, more diverse successor to ChartQA, purpose-built to remain unsaturated as frontier models continue improving on chart-specific benchmarks. ## Run ChartQAPro on Your Model Benchgen lets teams run ChartQAPro against their own model or agent versions on demand, tracking scores broken out by question type (including unanswerable questions) to catch regressions in real-world chart reasoning. ## Related Models & Benchmarks Explore related multimodal benchmarks like [MME-RealWorld](/benchmarks/zhang-et-al-xiamen-university/mme-realworld) and [MMBench-Video](/benchmarks/fang-et-al-shanghai-ai-laboratory/mmbench-video) to compare chart-specific reasoning with broader real-world perception and video understanding evaluation. ## Frequently Asked Questions

What is ChartQAPro?

ChartQAPro is a 1,948-question chart QA benchmark spanning 1,341 diverse real-world charts with multiple-choice, conversational, hypothetical, and unanswerable question formats, designed to restore difficulty lost to ChartQA's saturation.

What does a good ChartQAPro score look like?

Because scores drop sharply versus ChartQA (e.g., Claude Sonnet 3.5 fell from 90.5% to 55.81%), any model scoring consistently above 60% across all question types — including unanswerable ones — represents a genuinely strong result.

Who created ChartQAPro?

ChartQAPro was created by Ahmed Masry and collaborators at York University's IntelligentVis Lab.

Rank	Model	Score
1	claude-3-5-sonnet	55.81
2	gemini-2-0-flash	53.66
3	gemini-1-5-flash	45.97
4	gpt-4o	41.68
5	qwen2-vl-7b	37.17

ChartQAPro — Results

ChartQAPro — Leaderboard

ChartQAPro Leaderboard