WildBench
1 phaseActive
Real-world-query evaluation framework using WB-Score/WB-Reward, built from 1M+ human-chatbot logs by AI2; strongly correlated with Chatbot Arena Elo.
Overview
WildBench
Quick answer: WildBench is an automated evaluation framework from the Allen Institute for AI that benchmarks LLMs on 1,024 challenging, real-world user queries mined from over one million human-chatbot conversation logs, using two automated metrics (WB-Reward and WB-Score) that closely track human-voted Chatbot Arena Elo ratings.
At a Glance
What it tests: Real-world instruction-following quality across the messy, open-ended queries users actually send chatbots, not curated academic prompts.
Why it matters: WB-Score correlates at r=0.95 with human Elo ratings on hard tasks — higher than ArenaHard (0.91) or length-controlled AlpacaEval 2.0 (0.89) — making it a strong, cheaper-to-run proxy for arena-style human preference.
Known limitations: Automated judging (GPT-4-turbo-class judge model) inherits judge-model biases and length preferences, mitigated but not eliminated by task-specific checklists and a length-bias correction rule.
What WildBench Measures
WildBench's task set is curated from over a million real human-chatbot conversation logs, filtered down to 1,024 tasks that are difficult enough to meaningfully separate model quality. Each task ships with a task-specific checklist that the judge model uses to systematically evaluate a response, producing structured, interpretable explanations alongside the score rather than a bare number.
Two metrics come out of this pipeline. WB-Reward runs fine-grained pairwise comparisons against three baseline models spanning different capability tiers, classifying each matchup into one of five outcomes (much better / slightly better / tie / slightly worse / much worse); a length-bias correction converts "slightly better/worse" verdicts to ties when the winning response is disproportionately longer. WB-Score rates each response individually on a 1–10 scale (via GPT-4-turbo), rescaled by (Y-5)×2 so 5 represents a borderline response, producing a fast, single-pass metric.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Instruction following / open-ended chat quality |
| Metric | WB-Score (rescaled 1-10 judge score) and WB-Reward (pairwise vs. 3 baselines) |
| Number of tasks | 1,024 real-user tasks |
| Saturation | Medium — top models still separate meaningfully |
| Created by | Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, et al. (Allen Institute for AI) |
| Source paper | Lin et al. 2024 |
| GitHub | allenai/WildBench |
| Dataset | allenai/WildBench |
How WildBench Is Scored
WB-Score is the most commonly cited single number: an LLM judge scores each of the 1,024 responses from 1–10, and the raw score is rescaled by (Y-5)×2 so it centers around 0 for borderline responses. Scores in the +30 to +50 range are typical for strong frontier models; scores near 0 or negative indicate the model frequently produces unhelpful or malformed responses on real-world tasks. WB-Reward is reported as a win-rate-style percentage against the three fixed baseline models.
State-of-the-Art Results
Scores shift frequently as new frontier models are added. See the official leaderboard for current WB-Score/WB-Reward standings.
WildBench on Benchgen
No Benchgen results yet — be the first to run WildBench.
WildBench vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| WildBench | Real-user query quality (WB-Score/Reward) | 1,024 | medium |
| Arena Hard v2 | Curated hard prompts, pairwise judge | — | medium |
| AlpacaEval 2.0 | Length-controlled instruction win-rate | — | high |
| MT-Bench | Multi-turn conversation quality | — | high |
WildBench distinguishes itself by sourcing tasks directly from real deployed-chatbot logs rather than a curated prompt set, and by reporting the strongest correlation with human Chatbot Arena Elo among automated alternatives.
Run WildBench on Your Model
Benchgen lets teams run the WildBench task suite against their own model versions, tracking WB-Score and WB-Reward trends release over release — surfacing real-world quality regressions before they show up in production chat logs.