ZebraLogic
1 phaseActive
Logic grid puzzle benchmark from constraint satisfaction problems, testing non-monotonic reasoning at scale; created by AI2/UW researchers (ICML 2025).
Overview
ZebraLogic
Quick answer: ZebraLogic is a benchmark of 1,000 logic grid ("Zebra") puzzles built from constraint satisfaction problems, created by Allen Institute for AI and University of Washington researchers to test the scaling limits of LLM logical reasoning as puzzle complexity grows.
At a Glance
What it tests: Non-monotonic, multi-constraint deductive reasoning — filling in a grid of houses/attributes so every clue is simultaneously satisfied.
Why it matters: Puzzle difficulty is controllable and quantifiable (grid size × number of constraints), so ZebraLogic isolates how reasoning accuracy decays as search-space complexity grows, rather than just measuring a single difficulty tier.
Known limitations: Programmatically generated puzzles reduce contamination risk, but very large grids become intractable for any model, making the hardest tier a near-permanent floor rather than a target to saturate.
What ZebraLogic Measures
ZebraLogic frames each task as a classic Zebra/Einstein puzzle: N houses, each with M attributes (color, drink, pet, etc.), and a list of natural-language clues that constrain the arrangement. Solving it correctly requires holding multiple constraints in working memory and propagating deductions consistently — a single wrong inference cascades into a wrong final grid.
The paper's central finding, the "curse of complexity," is that accuracy collapses sharply as the search space grows (more houses, more constraints), and this holds even for very capable reasoning models with large inference-time compute budgets like o1 and DeepSeek-R1. The authors also test mitigations — Best-of-N sampling, backtracking, and self-verification prompts — showing partial but incomplete recovery.
Because puzzles are synthetically generated from CSP templates, new puzzle instances of arbitrary size can be produced on demand, making the benchmark resistant to memorization and useful for controlled scaling studies.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Reasoning (logical/constraint satisfaction) |
| Metric | Puzzle-level accuracy (all cells correct) and cell-level accuracy |
| Number of tasks | 1,000 puzzles (Easy and Hard splits by grid size) |
| Saturation | Low — accuracy declines sharply as complexity increases |
| Created by | Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, Yejin Choi |
| Source paper | Lin et al. 2025 |
| Dataset / Leaderboard | HuggingFace Space |
How ZebraLogic Is Scored
Each puzzle grid is scored two ways: cell-level accuracy (the fraction of individual cells filled in correctly) and the stricter puzzle-level accuracy (credit only if every cell in the grid is correct). Puzzles are split into Easy and Hard subsets by grid size (number of houses × attributes), and results are commonly reported separately for each subset since the "curse of complexity" makes aggregate scores hard to interpret without the split.
State-of-the-Art Results
Scores fluctuate by model release cadence and evaluation harness (sampling temperature, prompting strategy). See the official leaderboard for current standings across puzzle-size splits.
ZebraLogic on Benchgen
No Benchgen results yet — be the first to run ZebraLogic.
ZebraLogic vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| ZebraLogic | Constraint-satisfaction logic grid puzzles | 1,000 | low |
| ARC-AGI | Abstract visual pattern induction | — | low |
| MuSR | Multistep soft reasoning in narrative form | — | medium |
ZebraLogic isolates pure deductive/constraint reasoning without visual or narrative components, complementing ARC-AGI's pattern-induction focus and MuSR's narrative-embedded reasoning.
Run ZebraLogic on Your Model
Benchgen lets teams run ZebraLogic-style puzzle suites against their own model versions on demand, tracking cell- and puzzle-level accuracy across puzzle-size tiers over time — catching regressions in multi-constraint reasoning that a single aggregate score would hide.