CharXiv Reasoning

CharXiv Reasoning

Quick answer: CharXiv Reasoning is the complex reasoning sub-task of the CharXiv benchmark (Wang et al., 2024), which evaluates AI models on scientific chart understanding using figures extracted from real arXiv papers. The Reasoning sub-task asks multi-step analytical questions that require integrating visual data with scientific reasoning. Fugu Ultra scores 86.6% and Fugu scores 85.1% as of June 2026.

At a Glance

What it tests: Complex, multi-step reasoning about scientific charts and figures drawn from real arXiv papers — including extracting quantitative data, comparing trends, and drawing analytical conclusions.

Why it matters: Real-world AI assistants for scientific research must interpret figures and charts, not just text. CharXiv Reasoning provides a rigorous, contamination-resistant measure of this capability using actual research figures.

Known limitations: Scientific figures from arXiv may be niche and domain-specific; performance likely varies by scientific domain. Requires vision capability — text-only models cannot be evaluated.

What CharXiv Reasoning Measures

CharXiv extracts figures from recent arXiv preprints and generates two types of questions: descriptive (simple data reading) and reasoning (multi-step analytical). The Reasoning sub-task focuses on the harder analytical questions — for example, calculating percentage differences between trend lines, identifying crossover points, or drawing conclusions that require combining information from multiple chart elements.

By sourcing from recent arXiv preprints after model training cutoffs, CharXiv avoids contamination from training data. The reasoning sub-task is specifically challenging because it requires both accurate visual parsing and the ability to perform arithmetic or logical reasoning on extracted values.

Benchmark Specifications

State-of-the-Art Results

Scores sourced from Sakana AI's Fugu technical report, June 2026.