CharXiv Reasoning (No Tools)

CharXiv Reasoning (No Tools)

Quick answer: CharXiv Reasoning (No Tools) is the CharXiv benchmark's reasoning sub-task evaluated without Python code execution — models must reason about scientific charts from arXiv papers using only language, without the ability to write and run Python for numerical computation. Scores are slightly lower than the "with Python" variant. Claude Fable 5 leads the Inkling comparison set at 86.5%.

At a Glance

What it tests: A model's ability to answer complex reasoning questions about real scientific charts extracted from arXiv papers, using only its language reasoning capabilities (no Python tool use allowed).

Why it matters: Many frontier models support Python code execution as a reasoning tool. Evaluating CharXiv Reasoning without Python isolates pure multimodal language reasoning ability from programming-assisted computation. This variant shows which models can reason about charts through understanding alone — a more fundamental capability.

Difference from the "with Python" variant: The CharXiv Reasoning (with Python) variant allows models to write and execute Python code to extract or compute values from charts. The "no tools" variant does not permit this, making it harder for questions requiring numerical computation.

Benchmark Specifications

State-of-the-Art Results

Scores from Inkling model card (Thinking Machines Lab, July 2026). 6 models reported; Nemotron 3 Ultra, GLM 5.2, and DeepSeek V4 Pro not in this set.

Last updated 2026-07-16.