ChartQA — Results
| Rank | Model | Score |
|---|---|---|
| 1 | lfm2-5-vl-3b | 81.3 |
| 2 | north-micro-vision-instruct | 80.8 |
| 3 | claude-3-5-sonnet | 0.908 |
| 4 | llama-4-maverick | 0.9 |
| 5 | nova-pro | 0.892 |
| 6 | llama-4-scout | 0.888 |
| 7 | mistral-small-3-2-24b-instruct | 0.874 |
| 8 | nova-lite | 0.868 |
| 9 | gpt-4o | 0.857 |
| 10 | llama-3-2-90b-instruct | 0.855 |
| 11 | qwen2-5-omni-7b | 0.853 |
| 12 | gemma-3-27b | 0.78 |
| 13 | gemma-3-12b | 0.757 |
| 14 | gemma-3-4b | 0.688 |
ChartQA
1 phaseActive
32,700-question benchmark for visual and logical reasoning over charts — combining human-written and chart-summary-derived questions. Metric: relaxed accuracy.
Overview
ChartQA
Quick answer: ChartQA is a visual reasoning benchmark by Masry et al. (2022) with 9.6K human-written and 23.1K chart-summary-derived questions requiring logical and arithmetic reasoning over charts. Claude 3.5 Sonnet leads with 90.8% across 24 evaluated models.
What Does ChartQA Test?
ChartQA tests whether AI models can answer questions about charts that require reasoning, not just reading values off axes. Questions involve numerical comparisons, trend identification, and combining multiple chart elements — tasks that require understanding both visual structure and logical relationships.
| Question type | Example |
|---|---|
| Visual retrieval | "What is the value of X in 2020?" |
| Arithmetic reasoning | "By how much did X increase from 2018 to 2022?" |
| Comparison | "Which year had the highest value?" |
| Trend analysis | "Is the trend increasing or decreasing?" |
| Multi-bar/line reasoning | "Which category grew fastest?" |
The benchmark combines 9,608 human-written questions (harder, more diverse) and 23,101 questions generated from human-written chart summaries.
How Is ChartQA Scored?
ChartQA uses relaxed accuracy — numeric answers within 5% of the ground truth are accepted as correct to handle rounding variations. Scores are reported on a 0–1 scale.
Key Facts
| Property | Value |
|---|---|
| Published | March 2022 |
| Tasks | 32,698 questions |
| Human-written | 9,608 |
| Generated | 23,090 |
| Metric | Relaxed accuracy |
| Score range | 0–1 |
| Top model | Claude 3.5 Sonnet (0.908) |
| Models evaluated | 24 |
FAQ
What is ChartQA? ChartQA is a benchmark for visual and logical reasoning over charts, comprising 9.6K human-written questions and 23.1K questions generated from chart summaries. It tests whether AI models can perform numerical comparisons, trend analysis, and multi-element reasoning from chart images.
Who created ChartQA? ChartQA was created by Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque, published in March 2022 (arXiv 2203.10244).
What is relaxed accuracy in ChartQA? Relaxed accuracy accepts numeric predictions within 5% of the ground-truth value as correct, to handle minor rounding differences in answers extracted from charts.
What score does the best model achieve on ChartQA? Claude 3.5 Sonnet leads with 0.908 (90.8%), followed by Llama 4 Maverick at 0.900 and Qwen2.5 VL 72B Instruct at 0.895.