ScienceQA Visual
1 phaseActive
Image-grounded subset of ScienceQA testing multimodal science question answering with chain-of-thought explanations, created by UCLA, AI2, and Columbia researchers.
Overview
ScienceQA Visual
Quick answer: ScienceQA Visual is the image-grounded subset of ScienceQA, a large-scale multiple-choice science question benchmark spanning natural, social, and language science topics, where questions require reasoning over an accompanying image alongside text.
At a Glance
What it tests: Multimodal multiple-choice science QA at elementary-to-high-school level, requiring models to combine visual evidence (diagrams, photos, charts) with textual context and general knowledge.
Why it matters: ScienceQA was among the first large benchmarks to pair multiple-choice science questions with chain-of-thought explanations and lecture context, making it a standard testbed for both answer accuracy and reasoning quality.
Known limitations: Many questions are answerable from strong prior knowledge alone without the image, and frontier multimodal models are now largely saturated on the visual subset.
What ScienceQA Visual Measures
ScienceQA is a dataset of roughly 21,000 multimodal multiple-choice science questions spanning natural science, social science, and language science, annotated with grade level, topic/category/skill labels, and (for most questions) a lecture passage and explanation supporting chain-of-thought reasoning. The visual subset (ScienceQA-IMG) restricts evaluation to the ~10,332 questions that include an accompanying image, isolating the multimodal reasoning slice of the full dataset from the text-only portion.
Because each question is tagged with detailed metadata (subject, topic, skill, grade), ScienceQA supports fine-grained breakdowns of model performance — for example, comparing accuracy on biology diagrams versus geography maps — beyond a single aggregate accuracy number.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Multimodal (multiple-choice science QA) |
| Metric | Multiple-choice accuracy (%), often broken down by subject/grade/context type |
| Number of tasks | 10,332 image-grounded questions (of ~21,208 total questions in full ScienceQA) |
| Saturation | High — frontier multimodal LLMs exceed 90% accuracy on the visual subset |
| Created by | Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, Ashwin Kalyan |
| Source paper | Lu et al. 2022 (NeurIPS 2022) |
| Dataset | Hugging Face · GitHub |
How ScienceQA Visual Is Scored
Each question presents 2-5 answer options; the model's predicted option is compared against the ground-truth label to compute accuracy. Because many questions include lecture context and reference explanations, some evaluation setups also report explanation quality (e.g., via chain-of-thought generation scored against reference reasoning), though top-line leaderboard comparisons use plain multiple-choice accuracy.
State-of-the-Art Results
Scores fluctuate by model release cadence and evaluation harness (prompt template, use of chain-of-thought). See the official leaderboard for current standings.
ScienceQA Visual on Benchgen
No Benchgen results yet — be the first to run ScienceQA Visual.
ScienceQA Visual vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| ScienceQA Visual | Multimodal multiple-choice K-12 science QA | 10,332 | high |
| MedXpertQA | Expert-level multimodal medical QA | — | low |
| MME | General perception + cognition evaluation | — | high |
ScienceQA Visual's K-12 science focus sits at an easier difficulty tier than expert-level domain benchmarks like MedXpertQA, making it more useful for gauging general multimodal grounding than frontier specialist capability.
Run ScienceQA Visual on Your Model
Benchgen lets teams run the ScienceQA visual subset against their own model versions on demand, with breakdowns by subject, topic, and grade level to spot specific weak areas.
Related Models & Benchmarks
Explore related multimodal QA benchmarks like MedXpertQA and MME to compare domain-specific and general-purpose multimodal evaluation approaches.