# MuSR ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-756-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-medium-d29922) ![Created](https://img.shields.io/badge/Created-Oct%202023-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2310.16049) [![GitHub](https://img.shields.io/badge/GitHub-Zayne--sprague%2FMuSR-181717?logo=github&logoColor=white)](https://github.com/Zayne-sprague/MuSR) > **Quick answer:** MuSR (Multistep Soft Reasoning) is a University of Texas at Austin benchmark (ICLR 2024 Spotlight) that tests LLMs on multistep reasoning embedded in realistic free-text narratives — murder mysteries, object placement puzzles, and team allocation scenarios — specifically designed to challenge chain-of-thought prompting. ## At a Glance **What it tests:** Whether a model can hold and integrate scattered pieces of evidence spread across a long (roughly 1,000-word) natural-language narrative to reach a single correct conclusion, rather than solving a short, explicitly-structured logic puzzle. **Why it matters:** Most reasoning benchmarks use short, clean, explicitly-structured problems; MuSR instead embeds multistep logical and commonsense reasoning inside realistic prose, testing whether reasoning ability actually transfers to messier real-world-style text. **Known limitations:** MuSR's neurosymbolic synthetic-to-natural generation process means new instances can be produced as models improve, but the initial released dataset is fixed-size and could face contamination risk over time like any static benchmark. ## What MuSR Measures MuSR is built using a novel neurosymbolic synthetic-to-natural generation algorithm: the authors start from a structured symbolic reasoning tree (the ground-truth chain of deductions) and then generate a natural-language narrative around it that requires readers to combine commonsense knowledge with the embedded evidence to arrive at the same conclusion. This lets the benchmark scale to arbitrarily complex reasoning trees while keeping the surface text realistic and human-solvable. The dataset spans three domains: **murder mysteries** (identify the culprit from suspect narratives), **object placements** (track where an object ends up after a sequence of narrated events and character beliefs), and **team allocation** (assign people to tasks based on narrated skills and constraints). Each domain requires integrating "soft" evidence — commonsense inferences not explicitly spelled out — rather than purely formal logical deduction, distinguishing MuSR from harder-edged constraint-satisfaction benchmarks. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning / narrative-embedded multistep reasoning | | Metric | Accuracy | | Number of tasks | 756 total (250 murder mysteries, 256 object placements, 250 team allocation) | | Saturation | Medium — GPT-4 scored 80.4% on murder mysteries at publication, still below human ceiling | | Created by | Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, Greg Durrett (University of Texas at Austin) | | Source paper | [Sprague et al. 2024](https://arxiv.org/abs/2310.16049) (ICLR 2024 Spotlight) | | GitHub | [Zayne-sprague/MuSR](https://github.com/Zayne-sprague/MuSR) | ## How MuSR Is Scored Each of the three domains is scored independently as multiple-choice accuracy (e.g., "who is the murderer?" among named suspects). Because the domains test distinct reasoning styles (narrative deduction, spatial/belief tracking, and constraint-based allocation), results are typically reported per domain rather than only as one aggregate score. ## State-of-the-Art Results | Rank | Model | Score (murder mysteries) | Source | Date | |---|---|---|---|---| | — | GPT-4 (original paper baseline) | 80.4% | [Sprague et al. 2024](https://arxiv.org/abs/2310.16049) | 2023-10 | *The paper's own reported GPT-4 baseline on the murder mysteries domain; this specific model checkpoint does not have a corresponding Benchgen-tracked model entry, so it is shown for reference only and not as a synced leaderboard entry. See the [official GitHub repository](https://github.com/Zayne-sprague/MuSR) for the full per-model, per-domain breakdown.* ## MuSR on Benchgen > No Benchgen results yet — [be the first to run MuSR](/benchmarks/zayne-sprague-xi-ye-kaj-bostrom-swarat-chaudhuri-greg-durrett-university-of-texas-at-austin/musr). ## MuSR vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **MuSR** | Narrative-embedded multistep soft reasoning | 756 | medium | | [ZebraLogic](/benchmarks/bill-yuchen-lin-et-al-allen-institute-for-ai-university-of-washington/zebralogic) | Constraint-satisfaction logic grid puzzles | 1,000 | low | | [GPQA](/benchmarks/gpqa) | Graduate-level science question answering | — | low | MuSR is distinguished from purely formal reasoning benchmarks like ZebraLogic by requiring integration of unstated commonsense knowledge within realistic prose narratives, rather than testing pure constraint-satisfaction logic in isolation. ## Run MuSR on Your Model Benchgen lets teams evaluate their own model's narrative reasoning ability across MuSR's three domains, tracking domain-specific accuracy trends to catch regressions in commonsense-integrated multistep reasoning. ## Frequently Asked Questions

What is MuSR?

MuSR is a benchmark that tests multistep soft reasoning embedded in realistic free-text narratives across three domains: murder mysteries, object placements, and team allocation.

What does a good MuSR score look like?

At publication, GPT-4 scored 80.4% on the murder mysteries domain, still meaningfully below human-level performance, so scores in that range or higher represent strong but not yet saturated performance.

Who created MuSR?

MuSR was created by Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett at the University of Texas at Austin, published at ICLR 2024 as a Spotlight paper.

MuSR — Leaderboard

MuSR Leaderboard