LongBench v2 — Results
| Rank | Model | Score |
|---|---|---|
| 1 | qwen3-5-397b-a17b | 0.632 |
| 2 | qwen3-6-plus | 0.62 |
| 3 | nemotron-3-ultra-550b-a55b | 0.619 |
| 4 | minimax-m1-80k | 0.615 |
| 5 | kimi-k2-5 | 0.61 |
| 6 | mai-thinking-1 | 0.61 |
| 7 | minimax-m1-40k | 0.61 |
| 8 | mimo-v2-flash | 0.606 |
| 9 | qwen3-5-27b | 0.606 |
| 10 | qwen3-5-122b-a10b | 0.602 |
| 11 | qwen3-5-35b-a3b | 0.59 |
| 12 | qwen3-5-9b | 0.552 |
| 13 | qwen3-5-4b | 0.5 |
| 14 | deepseek-v3 | 0.487 |
| 15 | qwen3-5-2b | 0.387 |
| 16 | qwen3-5-08b | 0.261 |
LongBench v2
1 phaseActive
503 challenging long-context questions (8K–2M words) across single-doc QA, multi-doc QA, long ICL, dialogue history, code repos, and structured data. Metric: accuracy.
Overview
LongBench v2
Quick answer: LongBench v2 is a long-context benchmark by Bai et al. (2024) from Tsinghua University (THUDM) with 503 challenging multiple-choice questions requiring deep understanding and reasoning across contexts of 8K to 2M words. Qwen3.8 Max leads with 66.3% across 17 evaluated models.
What Does LongBench v2 Test?
LongBench v2 goes beyond simple retrieval — every question requires genuine reasoning over long documents. The benchmark covers six task categories:
| Category | Description |
|---|---|
| Single-document QA | Deep comprehension of a single long document |
| Multi-document QA | Synthesizing information across multiple documents |
| Long in-context learning | Few-shot learning from very long examples |
| Long dialogue history | Understanding context from extended conversations |
| Code repository understanding | Reasoning about large codebases |
| Long structured data understanding | Interpreting tables, spreadsheets, and structured content |
Context lengths range from 8K to 2M words, making LongBench v2 one of the most demanding tests of true long-context capability.
How Is LongBench v2 Scored?
All questions are multiple-choice, scored as correct or incorrect. The final score is the fraction of questions answered correctly, normalized to 0–1. Questions require multi-hop reasoning rather than simple lookup, resisting strategies that work on shorter contexts.
Key Facts
| Property | Value |
|---|---|
| Published | December 2024 |
| Tasks | 503 questions |
| Context range | 8K–2M words |
| Categories | 6 |
| Metric | Accuracy |
| Score range | 0–1 |
| Top model | Qwen3.8 Max (0.663) |
| Models evaluated | 17 |
FAQ
What is LongBench v2? LongBench v2 is a benchmark assessing LLMs on long-context problems requiring deep understanding and reasoning. It contains 503 challenging multiple-choice questions with contexts from 8K to 2M words across six task categories.
Who created LongBench v2? LongBench v2 was created by Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng and colleagues at Tsinghua University, published in December 2024 (arXiv 2412.15204).
How is LongBench v2 different from LongBench v1? LongBench v2 features harder questions requiring deeper reasoning (not just retrieval), longer context windows (up to 2M words), and a more diverse set of real-world task categories.
What score does the best model achieve on LongBench v2? Qwen3.8 Max currently leads with 0.663 (66.3%), followed by Qwen3.5-397B-A17B at 0.632 and Qwen3.6 Plus at 0.620.