MuSR

MuSR

Quick answer: MuSR (Multistep Soft Reasoning) is a University of Texas at Austin benchmark (ICLR 2024 Spotlight) that tests LLMs on multistep reasoning embedded in realistic free-text narratives — murder mysteries, object placement puzzles, and team allocation scenarios — specifically designed to challenge chain-of-thought prompting.

At a Glance

What it tests: Whether a model can hold and integrate scattered pieces of evidence spread across a long (roughly 1,000-word) natural-language narrative to reach a single correct conclusion, rather than solving a short, explicitly-structured logic puzzle.

Why it matters: Most reasoning benchmarks use short, clean, explicitly-structured problems; MuSR instead embeds multistep logical and commonsense reasoning inside realistic prose, testing whether reasoning ability actually transfers to messier real-world-style text.

Known limitations: MuSR's neurosymbolic synthetic-to-natural generation process means new instances can be produced as models improve, but the initial released dataset is fixed-size and could face contamination risk over time like any static benchmark.

What MuSR Measures

MuSR is built using a novel neurosymbolic synthetic-to-natural generation algorithm: the authors start from a structured symbolic reasoning tree (the ground-truth chain of deductions) and then generate a natural-language narrative around it that requires readers to combine commonsense knowledge with the embedded evidence to arrive at the same conclusion. This lets the benchmark scale to arbitrarily complex reasoning trees while keeping the surface text realistic and human-solvable.

The dataset spans three domains: murder mysteries (identify the culprit from suspect narratives), object placements (track where an object ends up after a sequence of narrated events and character beliefs), and team allocation (assign people to tasks based on narrated skills and constraints). Each domain requires integrating "soft" evidence — commonsense inferences not explicitly spelled out — rather than purely formal logical deduction, distinguishing MuSR from harder-edged constraint-satisfaction benchmarks.

Benchmark Specifications

How MuSR Is Scored

Each of the three domains is scored independently as multiple-choice accuracy (e.g., "who is the murderer?" among named suspects). Because the domains test distinct reasoning styles (narrative deduction, spatial/belief tracking, and constraint-based allocation), results are typically reported per domain rather than only as one aggregate score.

State-of-the-Art Results

The paper's own reported GPT-4 baseline on the murder mysteries domain; this specific model checkpoint does not have a corresponding Benchgen-tracked model entry, so it is shown for reference only and not as a synced leaderboard entry. See the official GitHub repository for the full per-model, per-domain breakdown.

MuSR on Benchgen

No Benchgen results yet — be the first to run MuSR.

MuSR vs Other Benchmarks

MuSR is distinguished from purely formal reasoning benchmarks like ZebraLogic by requiring integration of unstated commonsense knowledge within realistic prose narratives, rather than testing pure constraint-satisfaction logic in isolation.

Run MuSR on Your Model

Benchgen lets teams evaluate their own model's narrative reasoning ability across MuSR's three domains, tracking domain-specific accuracy trends to catch regressions in commonsense-integrated multistep reasoning.

Frequently Asked Questions