MT-Bench

MT-Bench

Quick answer: MT-Bench is an 80-question multi-turn conversation benchmark created by LMSYS in 2023, used to evaluate LLM instruction-following and conversational quality. A GPT-4 judge rates each response on a 1–10 scale across 8 domains (writing, roleplay, reasoning, math, coding, extraction, STEM, humanities). GPT-4 Turbo scores 9.32; GPT-3.5 Turbo scores 7.94. It is largely saturated for frontier models.

At a Glance

What it tests: Multi-turn conversational quality and instruction-following across 8 domains — the ability to maintain coherence across turns, follow complex instructions, and produce high-quality responses in diverse scenarios.

Why it matters: MT-Bench pioneered the "LLM-as-judge" evaluation paradigm and was among the first benchmarks to use GPT-4 to score model responses, enabling scalable evaluation of open-ended generation quality beyond accuracy on multiple-choice tasks.

Known limitations: MT-Bench has only 80 questions, making results noisy. Scores cluster at 8.5–9.3 for all frontier models. The judge model (GPT-4) may have biases toward its own response style. AlpacaEval 2.0 and Chatbot Arena Elo provide larger-scale alternatives.

What MT-Bench Measures

MT-Bench presents 80 two-turn conversations across 8 domains: writing, roleplay, reasoning, math, coding, extraction, STEM knowledge, and humanities. Each conversation requires the model to first answer an initial question, then follow up with a related second question that often builds on or refines the first response. This multi-turn design tests whether models can maintain coherence and context across turns — a real-world requirement for chat applications.

GPT-4 scores each response on a 1–10 scale using a structured rubric. The final MT-Bench score is the average across all 80 first-turn and second-turn responses. The benchmark was introduced alongside the Vicuna chatbot and helped establish that instruction-tuned models could rival GPT-3.5 in conversational quality.

Benchmark Specifications

How MT-Bench Is Scored

The GPT-4 judge receives the question, the model's response, and a grading rubric, then outputs a score from 1–10. The reported MT-Bench score is the average across all 160 individual turn scores (2 turns × 80 questions). Some labs report first-turn only or aggregate differently. Due to the small question count and judge variance, scores within 0.1–0.2 points should be considered statistically equivalent.

State-of-the-Art Results

Scores are GPT-4 judge ratings on a 1–10 scale. From published technical reports and the LMSYS MT-Bench leaderboard.

MT-Bench vs Other Benchmarks

MT-Bench is good for a quick chat quality baseline. AlpacaEval 2.0 and Chatbot Arena Elo provide more statistically reliable preference evaluation.

Run MT-Bench on Your Model

Benchgen lets teams run MT-Bench on their own fine-tuned models to quickly assess instruction-following quality before deployment.

Related Models & Benchmarks

• Top performers: GPT-4 Turbo · Claude 3 Opus · Claude 3.5 Sonnet
• Related benchmarks: AlpacaEval 2.0 · Arena Hard v2 · IFBench

Frequently Asked Questions

Benchmark definition paraphrased from Zheng et al. 2023. State-of-the-art scores sourced from published technical reports and attributed inline. Last updated 2026-07-23.