EQ-Bench
1 phaseActive
Emotional intelligence benchmark for LLMs; began as a 60-question quiz correlating with MMLU, evolved into multi-turn adversarial roleplay judged by an LLM panel.
Overview
EQ-Bench
Quick answer: EQ-Bench is an emotional intelligence benchmark for LLMs created by Samuel Paech; it began as a 60-question quiz predicting emotional-state intensity in dialogue (correlating r=0.97 with MMLU) and has since evolved into multi-turn adversarial roleplay scenarios judged on trust-building and social-cognition skill by an LLM panel.
At a Glance
What it tests: A model's ability to read emotional subtext, adapt to conflicting user preferences, and navigate socially difficult conversations without collapsing into pure validation or pure confrontation.
Why it matters: Real deployed chatbots must handle emotionally loaded, ambiguous conversations; EQ-Bench specifically stress-tests this rather than assuming helpfulness generalizes automatically from academic benchmarks.
Known limitations: Current scoring relies entirely on LLM-judge panels with no external human-rater validation, and both the judge panel and the simulated personas may encode judge-family stylistic biases.
What EQ-Bench Measures
The original 2023 EQ-Bench asked models to predict the intensity of specific emotions felt by characters in a dialogue, a format that produced highly repeatable results from just 60 English-language questions and correlated strongly (r=0.97) with broad-capability benchmarks like MMLU — evidence that the test was capturing general intelligence-adjacent signal, not a wholly separate trait.
The benchmark has since evolved substantially (current iteration: EQ-Bench 4) into a much richer format: a simulated "persona" user with hidden traits, sensitivities, and a private emotional state chats with the assessed model over 16 turns, and the transcript is scored by a panel of three LLM judges across six ability dimensions (bond & rapport, authenticity, attunement, meeting preferences & needs, emotion sensemaking, emotion management) using blind, bidirectional pairwise comparisons aggregated into an Elo rating via a Bradley–Terry fit. Roughly half the persona scenarios are adversarial by design (ego-threat navigation, power imbalance, reality distortion, social exclusion) specifically to surface failure modes that easy, cooperative scenarios would hide.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Reasoning / social-emotional intelligence |
| Metric | Elo rating from LLM-judge panel (current); originally a 0-10 emotion-intensity score |
| Number of tasks | 60 questions (original paper); 120 multi-turn personas (current EQ-Bench 4) |
| Saturation | Medium — top models separate meaningfully on adversarial scenarios |
| Created by | Samuel J. Paech |
| Source paper | Paech 2023 |
| GitHub | EQ-bench/EQ-Bench |
| Leaderboard | eqbench.com |
How EQ-Bench Is Scored
In the current version, three judge models (rotated across matchups, scoring blind and bidirectionally) each pick a winner and margin per ability dimension for every head-to-head matchup; a soft Bradley–Terry fit converts these fractional pairwise outcomes into a single Elo rating per model. A separate, non-scoring "behavioural traits" pass rates each transcript 0–10 on discrete traits for informational display only. Because scores are Elo-based and relative to the current model pool, absolute numbers are less meaningful than rank position and the score gap to nearby models.
State-of-the-Art Results
EQ-Bench's current Elo ratings shift as new judge-panel models and personas are added each release. See the live leaderboard for current standings.
EQ-Bench on Benchgen
No Benchgen results yet — be the first to run EQ-Bench.
EQ-Bench vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| EQ-Bench | Emotional/social intelligence in roleplay | 60-120 | medium |
| MT-Bench | General multi-turn conversation quality | — | high |
| WildBench | Real-world query quality | 1,024 | medium |
Unlike general chat-quality benchmarks, EQ-Bench specifically isolates emotional and social reasoning under adversarial persona conditions rather than measuring broad helpfulness.
Run EQ-Bench on Your Model
Benchgen lets teams run EQ-Bench-style roleplay scenarios against their own model versions, tracking Elo trends across releases to catch regressions in social-emotional handling that pure capability benchmarks wouldn't reveal.