# ScienceQA Visual ![Category](https://img.shields.io/badge/Category-Multimodal-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-10%2C332-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-high-d29922) ![Created](https://img.shields.io/badge/Created-Sep%202022-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2209.09513) [![GitHub](https://img.shields.io/badge/GitHub-Repo-181717?logo=github&logoColor=white)](https://github.com/lupantech/ScienceQA) > **Quick answer:** ScienceQA Visual is the image-grounded subset of ScienceQA, a large-scale multiple-choice science question benchmark spanning natural, social, and language science topics, where questions require reasoning over an accompanying image alongside text. ## At a Glance **What it tests:** Multimodal multiple-choice science QA at elementary-to-high-school level, requiring models to combine visual evidence (diagrams, photos, charts) with textual context and general knowledge. **Why it matters:** ScienceQA was among the first large benchmarks to pair multiple-choice science questions with chain-of-thought explanations and lecture context, making it a standard testbed for both answer accuracy and reasoning quality. **Known limitations:** Many questions are answerable from strong prior knowledge alone without the image, and frontier multimodal models are now largely saturated on the visual subset. ## What ScienceQA Visual Measures ScienceQA is a dataset of roughly 21,000 multimodal multiple-choice science questions spanning natural science, social science, and language science, annotated with grade level, topic/category/skill labels, and (for most questions) a lecture passage and explanation supporting chain-of-thought reasoning. The **visual subset** (ScienceQA-IMG) restricts evaluation to the ~10,332 questions that include an accompanying image, isolating the multimodal reasoning slice of the full dataset from the text-only portion. Because each question is tagged with detailed metadata (subject, topic, skill, grade), ScienceQA supports fine-grained breakdowns of model performance — for example, comparing accuracy on biology diagrams versus geography maps — beyond a single aggregate accuracy number. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Multimodal (multiple-choice science QA) | | Metric | Multiple-choice accuracy (%), often broken down by subject/grade/context type | | Number of tasks | 10,332 image-grounded questions (of ~21,208 total questions in full ScienceQA) | | Saturation | High — frontier multimodal LLMs exceed 90% accuracy on the visual subset | | Created by | Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, Ashwin Kalyan | | Source paper | [Lu et al. 2022](https://arxiv.org/abs/2209.09513) (NeurIPS 2022) | | Dataset | [Hugging Face](https://huggingface.co/datasets/derek-thomas/ScienceQA) · [GitHub](https://github.com/lupantech/ScienceQA) | ## How ScienceQA Visual Is Scored Each question presents 2-5 answer options; the model's predicted option is compared against the ground-truth label to compute accuracy. Because many questions include lecture context and reference explanations, some evaluation setups also report explanation quality (e.g., via chain-of-thought generation scored against reference reasoning), though top-line leaderboard comparisons use plain multiple-choice accuracy. ## State-of-the-Art Results > Scores fluctuate by model release cadence and evaluation harness (prompt template, use of chain-of-thought). See the [official leaderboard](https://github.com/lupantech/ScienceQA) for current standings. ## ScienceQA Visual on Benchgen > No Benchgen results yet — [be the first to run ScienceQA Visual](/benchmarks/pan-lu-et-al-ucla-allen-institute-for-ai-columbia-university/scienceqa-visual). ## ScienceQA Visual vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **ScienceQA Visual** | Multimodal multiple-choice K-12 science QA | 10,332 | high | | [MedXpertQA](/benchmarks/medxpertqa) | Expert-level multimodal medical QA | — | low | | [MME](/benchmarks/mme) | General perception + cognition evaluation | — | high | ScienceQA Visual's K-12 science focus sits at an easier difficulty tier than expert-level domain benchmarks like MedXpertQA, making it more useful for gauging general multimodal grounding than frontier specialist capability. ## Run ScienceQA Visual on Your Model Benchgen lets teams run the ScienceQA visual subset against their own model versions on demand, with breakdowns by subject, topic, and grade level to spot specific weak areas. ## Related Models & Benchmarks Explore related multimodal QA benchmarks like [MedXpertQA](/benchmarks/medxpertqa) and [MME](/benchmarks/mme) to compare domain-specific and general-purpose multimodal evaluation approaches. ## Frequently Asked Questions

What is ScienceQA Visual?

ScienceQA Visual is the image-grounded subset (ScienceQA-IMG) of the ScienceQA dataset, containing 10,332 multiple-choice science questions that require reasoning over an accompanying image.

What does a good ScienceQA Visual score look like?

Frontier multimodal LLMs now score above 90% accuracy on the visual subset, reflecting significant saturation since the benchmark's 2022 release.

Who created ScienceQA Visual?

ScienceQA was created by Pan Lu and collaborators at UCLA, the Allen Institute for AI, and Columbia University, published at NeurIPS 2022.

ScienceQA Visual — Phases

ScienceQA Visual