# WildBench ![Category](https://img.shields.io/badge/Category-Instruction%20Following-1f6feb) ![Metric](https://img.shields.io/badge/Metric-WB--Score-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-1%2C024-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-medium-d29922) ![Created](https://img.shields.io/badge/Created-Jun%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2406.04770) [![GitHub](https://img.shields.io/badge/GitHub-allenai%2FWildBench-181717?logo=github&logoColor=white)](https://github.com/allenai/WildBench) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/allenai/WildBench) > **Quick answer:** WildBench is an automated evaluation framework from the Allen Institute for AI that benchmarks LLMs on 1,024 challenging, real-world user queries mined from over one million human-chatbot conversation logs, using two automated metrics (WB-Reward and WB-Score) that closely track human-voted Chatbot Arena Elo ratings. ## At a Glance **What it tests:** Real-world instruction-following quality across the messy, open-ended queries users actually send chatbots, not curated academic prompts. **Why it matters:** WB-Score correlates at r=0.95 with human Elo ratings on hard tasks — higher than ArenaHard (0.91) or length-controlled AlpacaEval 2.0 (0.89) — making it a strong, cheaper-to-run proxy for arena-style human preference. **Known limitations:** Automated judging (GPT-4-turbo-class judge model) inherits judge-model biases and length preferences, mitigated but not eliminated by task-specific checklists and a length-bias correction rule. ## What WildBench Measures WildBench's task set is curated from over a million real human-chatbot conversation logs, filtered down to 1,024 tasks that are difficult enough to meaningfully separate model quality. Each task ships with a task-specific checklist that the judge model uses to systematically evaluate a response, producing structured, interpretable explanations alongside the score rather than a bare number. Two metrics come out of this pipeline. **WB-Reward** runs fine-grained pairwise comparisons against three baseline models spanning different capability tiers, classifying each matchup into one of five outcomes (much better / slightly better / tie / slightly worse / much worse); a length-bias correction converts "slightly better/worse" verdicts to ties when the winning response is disproportionately longer. **WB-Score** rates each response individually on a 1–10 scale (via GPT-4-turbo), rescaled by `(Y-5)×2` so 5 represents a borderline response, producing a fast, single-pass metric. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Instruction following / open-ended chat quality | | Metric | WB-Score (rescaled 1-10 judge score) and WB-Reward (pairwise vs. 3 baselines) | | Number of tasks | 1,024 real-user tasks | | Saturation | Medium — top models still separate meaningfully | | Created by | Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, et al. (Allen Institute for AI) | | Source paper | [Lin et al. 2024](https://arxiv.org/abs/2406.04770) | | GitHub | [allenai/WildBench](https://github.com/allenai/WildBench) | | Dataset | [allenai/WildBench](https://huggingface.co/datasets/allenai/WildBench) | ## How WildBench Is Scored WB-Score is the most commonly cited single number: an LLM judge scores each of the 1,024 responses from 1–10, and the raw score is rescaled by `(Y-5)×2` so it centers around 0 for borderline responses. Scores in the +30 to +50 range are typical for strong frontier models; scores near 0 or negative indicate the model frequently produces unhelpful or malformed responses on real-world tasks. WB-Reward is reported as a win-rate-style percentage against the three fixed baseline models. ## State-of-the-Art Results > Scores shift frequently as new frontier models are added. See the [official leaderboard](https://huggingface.co/spaces/allenai/WildBench) for current WB-Score/WB-Reward standings. ## WildBench on Benchgen > No Benchgen results yet — [be the first to run WildBench](/benchmarks/bill-yuchen-lin-et-al-allen-institute-for-ai/wildbench). ## WildBench vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **WildBench** | Real-user query quality (WB-Score/Reward) | 1,024 | medium | | [Arena Hard v2](/benchmarks/arena-hard-v2) | Curated hard prompts, pairwise judge | — | medium | | [AlpacaEval 2.0](/benchmarks/alpacaeval-2.0) | Length-controlled instruction win-rate | — | high | | [MT-Bench](/benchmarks/mt-bench) | Multi-turn conversation quality | — | high | WildBench distinguishes itself by sourcing tasks directly from real deployed-chatbot logs rather than a curated prompt set, and by reporting the strongest correlation with human Chatbot Arena Elo among automated alternatives. ## Run WildBench on Your Model Benchgen lets teams run the WildBench task suite against their own model versions, tracking WB-Score and WB-Reward trends release over release — surfacing real-world quality regressions before they show up in production chat logs. ## Frequently Asked Questions

What is WildBench?

WildBench is an automated LLM evaluation framework built from 1,024 challenging real-world user queries, scored with the WB-Score and WB-Reward metrics.

What does a good WildBench score look like?

WB-Scores in the +30 to +50 range typically indicate a strong frontier-tier model; scores near or below 0 indicate frequent unhelpful or malformed responses on real-world tasks.

Who created WildBench?

WildBench was created by Bill Yuchen Lin and collaborators at the Allen Institute for AI (AI2), published in 2024.

WildBench — Phases

WildBench