HellaSwag

HellaSwag

Quick answer: HellaSwag is a grounded commonsense inference benchmark of ~10,000 multiple-choice sentence-completion problems, created by Zellers et al. in 2019. Models must choose the most plausible continuation of a short activity description from four options. Human performance is 95.6%; frontier models now match or exceed this, making HellaSwag fully saturated as a frontier discriminator. It remains useful for evaluating smaller models.

At a Glance

What it tests: Grounded commonsense reasoning — specifically the ability to choose a physically and contextually plausible sentence continuation from four adversarially generated alternatives.

Why it matters: When introduced, HellaSwag exposed a major gap between BERT-class models (~48%) and human reasoning (~96%), establishing commonsense understanding as a key LLM capability. It became a universal baseline reported in virtually every major model release.

Known limitations: HellaSwag is completely saturated for frontier models (all GPT-4 class models score 95%+). It provides no signal for comparing frontier models and should only be used to evaluate smaller or older models on commonsense reasoning.

What HellaSwag Measures

HellaSwag uses adversarial filtering to create difficult sentence completion tasks. Each example presents a short activity description from WikiHow or ActivityNet Captions, then asks the model to choose which of four continuations is most plausible. The wrong answers are generated by a language model and then filtered to select the ones that fool a prior model — making the task much harder than randomly generated distractors.

The "Hella" in HellaSwag stands for "Harder Endings, Longer contexts, and Low-shot Activities." When introduced in 2019, it significantly outpaced BERT-class models that had saturated previous commonsense benchmarks. The task requires understanding physical actions, temporal sequences, and social conventions in everyday scenarios.

Benchmark Specifications

How HellaSwag Is Scored

Accuracy is the percentage of validation examples where the model chooses the correct continuation. Most evaluations use 0-shot or 10-shot prompting in a completion/likelihood format, where the model scores each candidate completion by its log-probability. Instruction-tuned models evaluated via generation may score differently than base models evaluated via log-likelihood.

State-of-the-Art Results

Human performance: 95.6%. All frontier models cluster near the human ceiling. Scores from published technical reports.

HellaSwag vs Other Benchmarks

HellaSwag is primarily useful for benchmarking smaller models (1B–13B parameters). For frontier model comparisons, GPQA Diamond and MMLU-Pro are more informative.

Run HellaSwag on Your Model

Benchgen lets teams evaluate their own model checkpoints on HellaSwag as part of a regression test suite alongside harder reasoning benchmarks.

Related Models & Benchmarks

• Top performers: GPT-4o · Claude 3 Opus · Claude 3.5 Sonnet
• Related benchmarks: MMLU · GPQA Diamond · BIG-Bench Hard

Frequently Asked Questions

Benchmark definition paraphrased from Zellers et al. 2019. State-of-the-art scores sourced from published technical reports and attributed inline. Last updated 2026-07-23.