Winogrande — Results
| Rank | Model | Score |
|---|---|---|
| 1 | gpt-4-0613 | 0.875 |
| 2 | mimo-v2-5-pro | 0.856 |
| 3 | command-r-plus | 0.854 |
| 4 | qwen2-72b-instruct | 0.851 |
| 5 | llama-3-1-nemotron-70b-instruct | 0.845 |
| 6 | gemma-2-27b | 0.837 |
| 7 | hermes-3-70b | 0.832 |
| 8 | qwen2-5-32b-instruct | 0.82 |
| 9 | phi-3-5-moe-instruct | 0.813 |
| 10 | qwen2-5-coder-32b-instruct | 0.808 |
| 11 | gemma-2-9b | 0.806 |
| 12 | mistral-nemo-instruct | 0.768 |
| 13 | ministral-8b-instruct | 0.753 |
| 14 | granite-3-3-8b-base | 0.744 |
| 15 | qwen2-5-coder-7b-instruct | 0.729 |
| 16 | gemma-3n-e4b-litert-preview | 0.717 |
| 17 | gemma-3n-e4b | 0.717 |
| 18 | phi-3-5-mini-instruct | 0.685 |
| 19 | phi-4-mini | 0.67 |
| 20 | gemma-3n-e2b-litert-preview | 0.668 |
| 21 | gemma-3n-e2b | 0.668 |
| 22 | ernie-4-5 | 0.513 |
Winogrande
1 phaseActive
Adversarial Winograd Schema Challenge at scale — 44,000 pronoun resolution problems testing machine commonsense reasoning. Metric: accuracy.
Overview
Winogrande
Quick answer: Winogrande (WinoGrande: An Adversarial Winograd Schema Challenge at Scale) is a large-scale commonsense reasoning benchmark of 44,000 pronoun resolution problems built with adversarial filtering to remove statistical shortcuts. GPT-4 leads with 87.5% across 22 evaluated models, though human performance sits at 94.0%.
What Does Winogrande Test?
Winogrande evaluates whether AI models truly understand commonsense relationships rather than exploiting spurious statistical patterns. Each problem presents a sentence with an ambiguous pronoun that must be resolved using world knowledge and reasoning about physical and social situations — an expanded, adversarially-filtered successor to the original Winograd Schema Challenge.
| Focus area | Examples |
|---|---|
| Pronoun resolution | Determining which entity a pronoun refers to |
| Commonsense reasoning | Using world knowledge to disambiguate sentences |
| Physical/social reasoning | Understanding cause, effect, and typical behavior |
| Bias robustness | Adversarial filtering removes shortcuts exploitable by language models |
How Is Winogrande Scored?
Each problem is a fill-in-the-blank sentence pair with two answer options; the model selects the correct referent. Accuracy is the fraction of correctly resolved problems, normalized to 0–1. AI2's adversarial filtering algorithm (AFLite) was used to reduce annotation artifacts and dataset-specific biases from the original 44K crowd-sourced problems.
Key Facts
| Property | Value |
|---|---|
| Released | 2019 |
| Problems | 44,000 |
| Metric | Accuracy |
| Score range | 0–1 |
| Top model | GPT-4 (0.875) |
| Human performance | 0.940 |
| Models evaluated | 22 |
FAQ
What is Winogrande? Winogrande is a large-scale adversarial commonsense reasoning benchmark of 44,000 pronoun resolution problems, designed to test whether AI models genuinely understand commonsense or merely exploit statistical shortcuts.
Who created Winogrande? Winogrande was created by researchers at the Allen Institute for AI (AI2) and the University of Washington — Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi — published in 2019.
How is Winogrande different from the original Winograd Schema Challenge? The original WSC contained only 273 expert-crafted problems. Winogrande scales this to 44,000 problems using crowdsourcing, then applies adversarial filtering (AFLite) to remove problems that language models could solve via spurious bias rather than genuine reasoning.
What score does the best model achieve on Winogrande? GPT-4 leads with 0.875 (87.5%), followed by Xiaomi's MiMo-V2.5-Pro at 0.856 and Cohere's Command R+ at 0.854 — all still below the human baseline of 0.940.
Is Winogrande saturated? Scores cluster in the 0.68–0.88 range for most modern models, well below human performance, indicating the adversarial filtering successfully preserved genuine difficulty even as the field has advanced.