AlpacaEval 2.0

AlpacaEval 2.0

Quick answer: AlpacaEval 2.0 is a preference benchmark of 805 diverse instructions, where GPT-4 Turbo judges whether a model's response is better than the GPT-4 Turbo (1106) reference response. The length-controlled (LC) win rate metric adjusts for length bias, making it more reliable than raw win rate. Claude 3.5 Sonnet and GPT-4o score ~52–57% LC win rate, meaning they are slightly preferred over GPT-4 Turbo in head-to-head comparisons.

At a Glance

What it tests: Open-ended instruction-following quality — whether a model's responses to diverse user queries are preferred by a GPT-4 annotator over the reference GPT-4 Turbo baseline.

Why it matters: AlpacaEval 2.0's length-controlled win rate is one of the most reliable automatic proxies for human preference on instruction-following tasks. It correlates strongly with human Chatbot Arena Elo while being far cheaper to compute.

Known limitations: GPT-4 as judge can favor GPT-4-style responses and may not reflect all human preferences. Single-turn only — does not evaluate multi-turn coherence. Prompt distribution (805 web/user questions) may not reflect specialized enterprise or coding tasks.

What AlpacaEval 2.0 Measures

AlpacaEval uses 805 instructions from the AlpacaFarm evaluation set, spanning diverse categories including creative writing, Q&A, brainstorming, coding help, and advice. For each instruction, a GPT-4 Turbo annotator compares the evaluated model's response against the reference GPT-4 Turbo (1106) response and picks the preferred one.

The key innovation of AlpacaEval 2.0 is the length-controlled (LC) win rate, which corrects for the well-documented tendency of LLM judges to prefer longer responses. The original win rate is regressed against response length to remove the length effect, producing a fairer comparison. LC win rate is reported as a percentage; 50% means the model is judged equally with GPT-4 Turbo, above 50% means it is preferred.

Benchmark Specifications

How AlpacaEval Is Scored

The LC win rate is computed by running pairwise comparisons between the evaluated model and the GPT-4 Turbo reference on all 805 instructions, then applying a linear regression adjustment to neutralize response length effects. A score above 50% means the model is preferred over GPT-4 Turbo 1106 on the average instruction. The leaderboard is maintained at tatsu-lab.github.io/alpaca_eval.

State-of-the-Art Results

Scores are LC win rate (%) vs. GPT-4 Turbo (gpt-4-1106-preview). Sourced from the official AlpacaEval 2 leaderboard.

AlpacaEval 2.0 vs Other Benchmarks

AlpacaEval 2.0 is best for comparing instruction-following quality across a broad range of everyday tasks. Arena Hard v2 focuses on harder, more adversarial instructions. MT-Bench adds multi-turn evaluation.

Run AlpacaEval on Your Model

Benchgen lets teams run AlpacaEval 2.0 on their own instruction-tuned models, track LC win rate over fine-tuning iterations, and compare against published baselines.

Related Models & Benchmarks

• Top performers: Claude 3.5 Sonnet · GPT-4o · Claude 3 Opus
• Related benchmarks: MT-Bench · Arena Hard v2 · IFBench

Frequently Asked Questions

Benchmark definition paraphrased from Dubois et al. 2024. State-of-the-art scores sourced from the official AlpacaEval 2 leaderboard. Last updated 2026-07-23.