QCalEval

QCalEval

Quick answer: QCalEval is the world's first benchmark for evaluating vision-language models on quantum computer calibration, created by Cao, Zhang et al. at NVIDIA (April 2026). It contains 243 entries drawn from real quantum processor data across 22 experiment families, with each entry testing a model across six structured question types — from technical plot description through to actionable calibration recommendations. The benchmark was developed because no standard evaluation existed for the growing class of AI models being deployed in quantum calibration workflows. The current best result is 74.7%, achieved by NVIDIA's domain-specific Ising-Calibration-1-35B model.

At a Glance

What it tests: Whether a vision-language model can correctly interpret quantum calibration experiment plots — scatter plots, line plots, and heatmaps of qubit measurements — and generate accurate scientific analysis across six structured question categories.

Why it matters: Quantum calibration is a critical bottleneck in building fault-tolerant quantum processors. AI agents are increasingly being deployed to automate this workflow, but until QCalEval there was no standardised way to measure whether a model actually produces reliable calibration analysis or just plausible-sounding text. QCalEval provides the first controlled evaluation of this capability against real QPU data from multiple hardware platforms.

Known limitations: The benchmark contains only 243 entries and covers two qubit modalities (superconducting qubits and neutral atoms). Performance on other modalities (trapped ions, quantum dots) and on out-of-distribution experimental configurations is not captured. Scores are assigned by LLM judges (GPT-5.4 and Gemini 3.1 Pro), introducing potential evaluator bias.

What QCalEval Measures

QCalEval tests a model's ability to analyse scientific plots from quantum computing calibration experiments. Each of the 243 benchmark entries consists of a 2D plot image (PNG) paired with six structured questions, each probing a distinct analytical capability:

• Q1 — Technical Description: Can the model accurately describe what the plot is showing — axes, data structure, and observed trends?
• Q2 — Experimental Conclusion: Does the model correctly identify the primary scientific conclusion from the experimental data?
• Q3 — Experimental Significance: Can it assess how significant or meaningful the observed result is in the context of quantum calibration?
• Q4 — Fit Quality Assessment: Does it correctly evaluate how well a fitted curve or model matches the data, identifying anomalies or poor fits?
• Q5 — Parameter Extraction: Can it extract specific numerical parameters or thresholds from the plot (e.g., qubit coherence times, gate fidelities)?
• Q6 — Experiment Success Classification: Does it correctly classify whether the calibration experiment passed, failed, or requires follow-up?

The benchmark spans 22 experiment families across 87 scenario types, covering measurements from real quantum hardware contributed by NVIDIA partners including IQM, Conductor Quantum / EeroQ, Fermilab, Infleqtion, and the National Physical Laboratory UK. Synthetic entries were also generated using NVIDIA's quantum simulation tools to improve coverage.

The six-question structure is intentional: each question targets a different cognitive step in the calibration workflow, from low-level visual parsing (Q1) through domain-specific reasoning (Q2–Q5) to an actionable pass/fail verdict (Q6). A model that performs well on Q1 but poorly on Q4 has useful pattern recognition but cannot reliably assess fit quality — a common failure mode for general-purpose VLMs.

Benchmark Specifications

How QCalEval Is Scored

Each answer is scored semantically using LLM judges (both GPT-5.4 and Gemini 3.1 Pro, with scores averaged between them) against a ground truth answer and a structured scoring rubric. Ground truth answers were initially written by human domain experts and then augmented into full responses by LLMs, with cross-validation between the two judge models and human expert review.

The final benchmark score is the mean semantic accuracy across all six question types and all 243 entries, expressed as a percentage. Per-question scores are also reported separately to identify capability profiles — frontier general-purpose models tend to score well on Q1 (technical description) but significantly lower on Q4 (fit quality) and Q6 (experiment success classification).

A score above 70% is considered strong performance on this benchmark. Frontier general-purpose models (Gemini 3.1 Pro, Claude Opus 4.6, GPT-5.4) score in the 64–73% range zero-shot, while the domain-specific Ising-Calibration-1-35B reaches 74.7%.

State-of-the-Art Results

Scores sourced from the QCalEval paper and NVIDIA Ising blog post. All results are zero-shot evaluations. Scoring uses LLM-as-judge (average of GPT-5.4 and Gemini 3.1 Pro judges).

QCalEval on Benchgen

No Benchgen results yet — be the first to run QCalEval.

QCalEval vs Other Benchmarks

QCalEval is unique in its domain specificity: it is the only benchmark targeting quantum calibration workflows with real QPU data. General scientific reasoning benchmarks like MMLU-Pro include science questions but do not capture the visual, plot-interpretation component that matters for calibration automation. For teams building AI-powered quantum workflows, QCalEval is the only available proxy for production calibration quality.

Run QCalEval on Your Model

Benchgen lets teams run QCalEval against their own model or fine-tuned variant with version-controlled benchmark history. Rather than relying on the single snapshot published in the NVIDIA paper, running QCalEval through Benchgen lets you track how calibration performance evolves across model updates — and detect regressions before a fine-tuned model is deployed in an active calibration workflow.

Related Models & Benchmarks

• Top performers: Ising-Calibration-1-35B · Gemini 3.1 Pro · Claude Opus 4.6
• Related benchmarks: MMLU-Pro · SciCode

Frequently Asked Questions