# AlpacaEval 2.0 ![Category](https://img.shields.io/badge/Category-Preference-1f6feb) ![Metric](https://img.shields.io/badge/Metric-LC%20Win%20Rate-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-805-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Medium-d29922) ![Created](https://img.shields.io/badge/Created-2024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202404.04475-b31b1b)](https://arxiv.org/abs/2404.04475) [![GitHub](https://img.shields.io/badge/GitHub-tatsu--lab%2Falpaca__eval-181717?logo=github&logoColor=white)](https://github.com/tatsu-lab/alpaca_eval) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/tatsu-lab/alpaca_eval) [![Leaderboard](https://img.shields.io/badge/Leaderboard-tatsu--lab.github.io-0969da)](https://tatsu-lab.github.io/alpaca_eval/) > **Quick answer:** AlpacaEval 2.0 is a preference benchmark of 805 diverse instructions, where GPT-4 Turbo judges whether a model's response is better than the GPT-4 Turbo (1106) reference response. The **length-controlled (LC) win rate** metric adjusts for length bias, making it more reliable than raw win rate. Claude 3.5 Sonnet and GPT-4o score ~52–57% LC win rate, meaning they are slightly preferred over GPT-4 Turbo in head-to-head comparisons. ## At a Glance **What it tests:** Open-ended instruction-following quality — whether a model's responses to diverse user queries are preferred by a GPT-4 annotator over the reference GPT-4 Turbo baseline. **Why it matters:** AlpacaEval 2.0's length-controlled win rate is one of the most reliable automatic proxies for human preference on instruction-following tasks. It correlates strongly with human Chatbot Arena Elo while being far cheaper to compute. **Known limitations:** GPT-4 as judge can favor GPT-4-style responses and may not reflect all human preferences. Single-turn only — does not evaluate multi-turn coherence. Prompt distribution (805 web/user questions) may not reflect specialized enterprise or coding tasks. ## What AlpacaEval 2.0 Measures AlpacaEval uses 805 instructions from the AlpacaFarm evaluation set, spanning diverse categories including creative writing, Q&A, brainstorming, coding help, and advice. For each instruction, a GPT-4 Turbo annotator compares the evaluated model's response against the reference GPT-4 Turbo (1106) response and picks the preferred one. The key innovation of AlpacaEval 2.0 is the **length-controlled (LC) win rate**, which corrects for the well-documented tendency of LLM judges to prefer longer responses. The original win rate is regressed against response length to remove the length effect, producing a fairer comparison. LC win rate is reported as a percentage; 50% means the model is judged equally with GPT-4 Turbo, above 50% means it is preferred. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Preference / instruction-following | | Metric | Length-controlled (LC) win rate (%) vs. GPT-4 Turbo 1106 | | Number of tasks | 805 instructions | | Reference model | GPT-4 Turbo (gpt-4-1106-preview) = ~50% by definition | | Judge | GPT-4 Turbo | | Saturation | Medium (strong models cluster 40–57%) | | Created by | Dubois et al. (Stanford) | | Source paper | [Length-Controlled AlpacaEval (2024)](https://arxiv.org/abs/2404.04475) | | GitHub | [tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval) | | Dataset | [HuggingFace — tatsu-lab/alpaca_eval](https://huggingface.co/datasets/tatsu-lab/alpaca_eval) | ## How AlpacaEval Is Scored The LC win rate is computed by running pairwise comparisons between the evaluated model and the GPT-4 Turbo reference on all 805 instructions, then applying a linear regression adjustment to neutralize response length effects. A score above 50% means the model is preferred over GPT-4 Turbo 1106 on the average instruction. The leaderboard is maintained at [tatsu-lab.github.io/alpaca_eval](https://tatsu-lab.github.io/alpaca_eval/). ## State-of-the-Art Results | Rank | Model | Score | Source | Date | |---|---|---|---|---| | 1 | [Claude 3.5 Sonnet](/models/anthropic/claude-3-5-sonnet) | 52.4% | [AlpacaEval 2 leaderboard](https://tatsu-lab.github.io/alpaca_eval/) | 2024-06 | | 1 | [GPT-4o](/models/openai/gpt-4o) | 51.3% | [AlpacaEval 2 leaderboard](https://tatsu-lab.github.io/alpaca_eval/) | 2024-05 | | 3 | [GPT-4 Turbo](/models/openai/gpt-4-turbo) | 50.0% | [AlpacaEval 2 leaderboard](https://tatsu-lab.github.io/alpaca_eval/) | 2023-11 | | 4 | [Claude 3 Opus](/models/anthropic/claude-3-opus) | 40.5% | [AlpacaEval 2 leaderboard](https://tatsu-lab.github.io/alpaca_eval/) | 2024-03 | | 5 | [Llama 3.1 405B](/models/meta/llama-3-1-405b) | 39.3% | [AlpacaEval 2 leaderboard](https://tatsu-lab.github.io/alpaca_eval/) | 2024-07 | | 6 | [Claude 3 Sonnet](/models/anthropic/claude-3-sonnet) | 30.4% | [AlpacaEval 2 leaderboard](https://tatsu-lab.github.io/alpaca_eval/) | 2024-03 | | 7 | [Llama 3.1 70B](/models/meta/llama-3-1-70b) | 27.0% | [AlpacaEval 2 leaderboard](https://tatsu-lab.github.io/alpaca_eval/) | 2024-07 | | 8 | [Gemini 1.5 Pro](/models/google/gemini-1-5-pro) | 26.3% | [AlpacaEval 2 leaderboard](https://tatsu-lab.github.io/alpaca_eval/) | 2024-02 | *Scores are LC win rate (%) vs. GPT-4 Turbo (gpt-4-1106-preview). Sourced from the official AlpacaEval 2 leaderboard.* ## AlpacaEval 2.0 vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **AlpacaEval 2.0** | Single-turn instruction following (LC win rate) | 805 | Medium | | [MT-Bench](/benchmarks/zheng-et-al-lmsys/mt-bench) | Multi-turn chat quality (GPT-4 score 1–10) | 80 | High | | [Arena Hard v2](/benchmarks/lmarena/arena-hard-v2) | Hard instruction following (win rate) | 500 | Low | AlpacaEval 2.0 is best for comparing instruction-following quality across a broad range of everyday tasks. Arena Hard v2 focuses on harder, more adversarial instructions. MT-Bench adds multi-turn evaluation. ## Run AlpacaEval on Your Model Benchgen lets teams run AlpacaEval 2.0 on their own instruction-tuned models, track LC win rate over fine-tuning iterations, and compare against published baselines. ## Related Models & Benchmarks - Top performers: [Claude 3.5 Sonnet](/models/anthropic/claude-3-5-sonnet) · [GPT-4o](/models/openai/gpt-4o) · [Claude 3 Opus](/models/anthropic/claude-3-opus) - Related benchmarks: [MT-Bench](/benchmarks/zheng-et-al-lmsys/mt-bench) · [Arena Hard v2](/benchmarks/lmarena/arena-hard-v2) · [IFBench](/benchmarks/allen-institute-for-ai/ifbench) ## Frequently Asked Questions

What is AlpacaEval 2.0?

AlpacaEval 2.0 is a preference benchmark of 805 instructions where GPT-4 Turbo judges whether a model's response beats the GPT-4 Turbo (1106) reference. The length-controlled (LC) win rate metric corrects for length bias, making it a reliable proxy for human preference on instruction-following tasks.

What does a good AlpacaEval 2.0 score look like?

The reference (GPT-4 Turbo 1106) scores ~50%. Scores above 50% mean the model is preferred over GPT-4 Turbo. Top models (Claude 3.5 Sonnet, GPT-4o) score 51–57%. Scores below 30% indicate meaningfully weaker instruction-following than GPT-4 Turbo.

Who created AlpacaEval?

AlpacaEval 2.0 was created by Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto at Stanford. See [Dubois et al. 2024](https://arxiv.org/abs/2404.04475).

What is length-controlled (LC) win rate?

LC win rate applies a regression correction to remove the effect of response length on GPT-4's judgment. Without this correction, models that generate longer responses tend to win more pairwise comparisons, inflating their raw win rate. The LC win rate gives a fairer comparison independent of verbosity.

--- *Benchmark definition paraphrased from [Dubois et al. 2024](https://arxiv.org/abs/2404.04475). State-of-the-art scores sourced from the [official AlpacaEval 2 leaderboard](https://tatsu-lab.github.io/alpaca_eval/). Last updated 2026-07-23.*

Rank	Model	Score
1	claude-3-5-sonnet	52.4
2	gpt-4o	51.3
3	gpt-4-turbo	50
4	claude-3-opus	40.5
5	llama-3-1-405b-instruct	39.3
6	claude-3-sonnet	30.4
7	llama-3-1-70b-instruct	27
8	gemini-1-5-pro	26.3

AlpacaEval 2.0 — Results

AlpacaEval 2.0 — Phases

AlpacaEval 2.0