Arena-Hard-v2

Arena-Hard-v2

Quick answer: Arena-Hard-v2 (Li et al., 2024, v2 released April 2025) is an automatic LLM evaluation framework using 500 challenging real-world user queries plus 250 creative writing prompts sourced from Chatbot Arena. Models are judged by frontier LLMs (GPT-4.1 and/or Gemini-2.5) acting as proxies for human preference, making it one of the highest-correlation automatic proxies for Chatbot Arena rankings.

At a Glance

What it tests: Open-ended instruction following, complex reasoning, and creative writing ability using hard prompts derived from real Chatbot Arena conversations.

Why it matters: A high Arena-Hard-v2 score strongly predicts how well a model will rank against humans in Chatbot Arena pairwise battles, providing a cheap, fast, and reproducible signal before deploying to live arenas.

Known limitations: Style-control is applied to reduce verbosity bias, but LLM judges still carry systematic preferences (e.g., preferring longer or more structured answers), and the 750-prompt set may not capture all capability dimensions.

What Arena-Hard-v2 Measures

Arena-Hard-v2 selects hard prompts from real user conversations on Chatbot Arena that are most likely to generate disagreements between strong frontier models. The v2.0-Preview set contains 500 open-ended software engineering, math, and reasoning prompts alongside 250 creative writing prompts, all sourced from actual user interactions. This ensures the tasks reflect genuine user intent rather than artificially constructed test items.

Evaluation is done by pitting each model's response head-to-head against a baseline model using GPT-4.1 and/or Gemini-2.5 as LLM judges. The win rate — the percentage of prompts on which a model beats the baseline — is the primary score. Style control corrects for surface-level differences like response length and markdown density so scores reflect quality rather than formatting choices.

The benchmark was designed to address the high cost and latency of human Chatbot Arena ratings, while maintaining maximum alignment with human preference rankings. Empirically, Arena-Hard-v2 achieves the highest Spearman correlation to human preference among popular open-ended benchmarks at the time of publication.

Benchmark Specifications

How Arena-Hard-v2 Is Scored

Each model's response is compared pairwise against a fixed baseline model (gpt-4.1 by default). An LLM judge rates whether the model's answer wins, ties, or loses against the baseline for each prompt. The win rate is the percentage of prompts on which the evaluated model is judged to be better. A score of 50% means the model performs roughly at baseline level; higher scores indicate the model beats the baseline on more prompts. Style control (markdown and length normalization) is applied by default to prevent formatting artifacts from inflating scores.

State-of-the-Art Results

FAQ

What is Arena-Hard-v2? Arena-Hard-v2 is an automatic LLM benchmarking tool that uses 750 hard prompts sourced from real Chatbot Arena conversations to estimate how a model would rank in human preference evaluations.

How does Arena-Hard-v2 differ from Chatbot Arena? Chatbot Arena collects live human votes through pairwise comparisons, which is accurate but slow and expensive. Arena-Hard-v2 replaces human judges with frontier LLMs (GPT-4.1 or Gemini-2.5) to produce rankings that highly correlate with human preference at a fraction of the cost.

What is a good Arena-Hard-v2 score? A score above 70% indicates performance significantly better than the GPT-4.1 baseline. As of April 2025, only a handful of frontier reasoning models (o3, o4-mini-high, Gemini 2.5 Pro) exceed 75%.