# QCalEval ![Category](https://img.shields.io/badge/Category-Scientific%20Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Semantic%20Accuracy%20%25-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-243-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-Apr%202026-555) [![Paper](https://img.shields.io/badge/Paper-NVIDIA%20Research-76b900)](https://research.nvidia.com/publication/2026-04_qcaleval-benchmarking-vision-language-models-quantum-calibration-plot) [![GitHub](https://img.shields.io/badge/GitHub-nvidia%2FQCalEval-181717?logo=github&logoColor=white)](https://github.com/nvidia/QCalEval) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/nvidia/QCalEval) > **Quick answer:** QCalEval is the world's first benchmark for evaluating vision-language models on quantum computer calibration, created by Cao, Zhang et al. at NVIDIA (April 2026). It contains 243 entries drawn from real quantum processor data across 22 experiment families, with each entry testing a model across six structured question types — from technical plot description through to actionable calibration recommendations. The benchmark was developed because no standard evaluation existed for the growing class of AI models being deployed in quantum calibration workflows. The current best result is 74.7%, achieved by NVIDIA's domain-specific Ising-Calibration-1-35B model. ## At a Glance **What it tests:** Whether a vision-language model can correctly interpret quantum calibration experiment plots — scatter plots, line plots, and heatmaps of qubit measurements — and generate accurate scientific analysis across six structured question categories. **Why it matters:** Quantum calibration is a critical bottleneck in building fault-tolerant quantum processors. AI agents are increasingly being deployed to automate this workflow, but until QCalEval there was no standardised way to measure whether a model actually produces reliable calibration analysis or just plausible-sounding text. QCalEval provides the first controlled evaluation of this capability against real QPU data from multiple hardware platforms. **Known limitations:** The benchmark contains only 243 entries and covers two qubit modalities (superconducting qubits and neutral atoms). Performance on other modalities (trapped ions, quantum dots) and on out-of-distribution experimental configurations is not captured. Scores are assigned by LLM judges (GPT-5.4 and Gemini 3.1 Pro), introducing potential evaluator bias. ## What QCalEval Measures QCalEval tests a model's ability to analyse scientific plots from quantum computing calibration experiments. Each of the 243 benchmark entries consists of a 2D plot image (PNG) paired with six structured questions, each probing a distinct analytical capability: - **Q1 — Technical Description:** Can the model accurately describe what the plot is showing — axes, data structure, and observed trends? - **Q2 — Experimental Conclusion:** Does the model correctly identify the primary scientific conclusion from the experimental data? - **Q3 — Experimental Significance:** Can it assess how significant or meaningful the observed result is in the context of quantum calibration? - **Q4 — Fit Quality Assessment:** Does it correctly evaluate how well a fitted curve or model matches the data, identifying anomalies or poor fits? - **Q5 — Parameter Extraction:** Can it extract specific numerical parameters or thresholds from the plot (e.g., qubit coherence times, gate fidelities)? - **Q6 — Experiment Success Classification:** Does it correctly classify whether the calibration experiment passed, failed, or requires follow-up? The benchmark spans 22 experiment families across 87 scenario types, covering measurements from real quantum hardware contributed by NVIDIA partners including IQM, Conductor Quantum / EeroQ, Fermilab, Infleqtion, and the National Physical Laboratory UK. Synthetic entries were also generated using NVIDIA's quantum simulation tools to improve coverage. The six-question structure is intentional: each question targets a different cognitive step in the calibration workflow, from low-level visual parsing (Q1) through domain-specific reasoning (Q2–Q5) to an actionable pass/fail verdict (Q6). A model that performs well on Q1 but poorly on Q4 has useful pattern recognition but cannot reliably assess fit quality — a common failure mode for general-purpose VLMs. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Scientific reasoning / Visual QA | | Metric | Semantic accuracy % (LLM-as-judge, averaged over Q1–Q6) | | Number of benchmark entries | 243 | | QA pairs total | 1,458 (6 per entry) | | Plot types | Scatter, line, heatmap (PNG) | | Experiment families | 22 | | Scenario types | 87 | | QPU modalities | Superconducting qubits, neutral atoms | | Few-shot variant | 236 entries, 3 QA pairs each (708 total) | | Saturation | Low (best result: 74.7%) | | Dataset license | CC BY 4.0 | | Created by | Shuxiang Cao, Zijian Zhang et al. (NVIDIA) | | Source paper | [QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding](https://research.nvidia.com/publication/2026-04_qcaleval-benchmarking-vision-language-models-quantum-calibration-plot) | | GitHub | [nvidia/QCalEval](https://github.com/nvidia/QCalEval) | | Dataset | [nvidia/QCalEval on HuggingFace](https://huggingface.co/datasets/nvidia/QCalEval) | ## How QCalEval Is Scored Each answer is scored semantically using LLM judges (both GPT-5.4 and Gemini 3.1 Pro, with scores averaged between them) against a ground truth answer and a structured scoring rubric. Ground truth answers were initially written by human domain experts and then augmented into full responses by LLMs, with cross-validation between the two judge models and human expert review. The final benchmark score is the mean semantic accuracy across all six question types and all 243 entries, expressed as a percentage. Per-question scores are also reported separately to identify capability profiles — frontier general-purpose models tend to score well on Q1 (technical description) but significantly lower on Q4 (fit quality) and Q6 (experiment success classification). A score above 70% is considered strong performance on this benchmark. Frontier general-purpose models (Gemini 3.1 Pro, Claude Opus 4.6, GPT-5.4) score in the 64–73% range zero-shot, while the domain-specific Ising-Calibration-1-35B reaches 74.7%. ## State-of-the-Art Results | Rank | Model | Mean | Q1 | Q2 | Q3 | Q4 | Q5 | Q6 | Source | Date | |---|---|---|---|---|---|---|---|---|---|---| | 1 | [Ising-Calibration-1-35B](/models/nvidia/ising-calibration-1-35b-a3b) | 74.7% | 87.0% | 67.1% | 64.7% | 90.5% | 62.5% | 75.3% | [NVIDIA HF model card](https://huggingface.co/nvidia/Ising-Calibration-1-35B-A3B) | 2026-04 | | 2 | [Gemini 3.1 Pro](/models/google/gemini-3-1-pro) | 72.3% | 88.5% | 57.2% | 61.1% | 84.4% | 64.7% | 71.2% | [NVIDIA blog](https://developer.nvidia.com/blog/nvidia-ising-introduces-ai-powered-workflows-to-build-fault-tolerant-quantum-systems/) | 2026-04 | | 3 | [Claude Opus 4.6](/models/anthropic/claude-opus-4-6) | 67.8% | 90.0% | 49.0% | 65.5% | 76.1% | 64.3% | 60.0% | [NVIDIA blog](https://developer.nvidia.com/blog/nvidia-ising-introduces-ai-powered-workflows-to-build-fault-tolerant-quantum-systems/) | 2026-04 | | 4 | [GPT-5.4](/models/openai/gpt-5-4) | 64.6% | 90.9% | 52.7% | 63.7% | 54.7% | 64.3% | 51.3% | [NVIDIA blog](https://developer.nvidia.com/blog/nvidia-ising-introduces-ai-powered-workflows-to-build-fault-tolerant-quantum-systems/) | 2026-04 | *Scores sourced from the QCalEval paper and NVIDIA Ising blog post. All results are zero-shot evaluations. Scoring uses LLM-as-judge (average of GPT-5.4 and Gemini 3.1 Pro judges).* ## QCalEval on Benchgen > No Benchgen results yet — [be the first to run QCalEval](/benchmarks/qcaleval). ## QCalEval vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **QCalEval** | Quantum calibration plot analysis (VLM) | 243 | Low | | [MMLU-Pro](/benchmarks/mmlu-pro) | Graduate-level reasoning across 14 subjects | 12,032 | Medium | | [SciCode](/benchmarks/scicode) | Scientific coding across research domains | — | Low | QCalEval is unique in its domain specificity: it is the only benchmark targeting quantum calibration workflows with real QPU data. General scientific reasoning benchmarks like MMLU-Pro include science questions but do not capture the visual, plot-interpretation component that matters for calibration automation. For teams building AI-powered quantum workflows, QCalEval is the only available proxy for production calibration quality. ## Run QCalEval on Your Model Benchgen lets teams run QCalEval against their own model or fine-tuned variant with version-controlled benchmark history. Rather than relying on the single snapshot published in the NVIDIA paper, running QCalEval through Benchgen lets you track how calibration performance evolves across model updates — and detect regressions before a fine-tuned model is deployed in an active calibration workflow. ## Related Models & Benchmarks - Top performers: [Ising-Calibration-1-35B](/models/nvidia/ising-calibration-1-35b-a3b) · [Gemini 3.1 Pro](/models/google/gemini-3-1-pro) · [Claude Opus 4.6](/models/anthropic/claude-opus-4-6) - Related benchmarks: [MMLU-Pro](/benchmarks/mmlu-pro) · [SciCode](/benchmarks/scicode) ## Frequently Asked Questions

What is QCalEval?

QCalEval is the world's first benchmark for evaluating vision-language models on quantum computer calibration tasks, created by NVIDIA in April 2026. It contains 243 entries from real quantum hardware experiments, each pairing a 2D calibration plot with six structured questions testing a model's ability to interpret, reason, and generate actionable calibration recommendations.

What does a good QCalEval score look like?

The benchmark average for frontier general-purpose models is 64–73%. A score above 70% indicates strong calibration plot reasoning. The current best result is 74.7% by NVIDIA's domain-specific Ising-Calibration-1-35B. The benchmark is unsaturated — significant room for improvement remains across all question types.

Who created QCalEval?

QCalEval was created by Shuxiang Cao, Zijian Zhang and colleagues at NVIDIA, in collaboration with quantum hardware partners including IQM, Conductor Quantum, Fermilab, Infleqtion, and the National Physical Laboratory UK. The benchmark was released alongside the Ising-Calibration-1-35B model in April 2026.

Is the QCalEval dataset publicly available?

Yes. The dataset is available under CC BY 4.0 on HuggingFace at nvidia/QCalEval, and evaluation scripts are available at github.com/nvidia/QCalEval.

Why is Q4 (Fit Quality Assessment) the most discriminating question?

Q4 shows the largest spread between models: Ising-Calibration-1-35B scores 90.5% while GPT-5.4 scores only 54.7%. Fit quality assessment requires domain-specific judgment about whether a mathematical model appropriately fits experimental data — a skill that general-purpose VLMs have not been trained to apply in the quantum calibration context.

Rank	Model	Score
1	ising-calibration-1-35b	74.7
2	ising-calibration-1-35b	74.7
3	gemini-3-1-pro	72.3
4	claude-opus-4-6	67.8
5	gpt-5-4	64.6

QCalEval — Results

QCalEval — Evaluations