CRAG — Results
| Rank | Model | Score |
|---|---|---|
| 1 | gpt-4-turbo | 43.6 |
CRAG
1 phaseActive
Meta's 4,409-question RAG benchmark with mock web/KG search APIs across 5 domains, penalizing hallucinations more than missing answers; basis of KDD Cup 2024.
Overview
CRAG
Quick answer: CRAG (Comprehensive RAG Benchmark) is a Meta-created factual question-answering benchmark of 4,409 QA pairs across five domains, paired with mock web-search and knowledge-graph APIs to simulate realistic retrieval-augmented generation, and scored with a truthfulness metric that penalizes hallucinated answers more heavily than missing ones. It laid the groundwork for the 2024 KDD Cup challenge.
At a Glance
What it tests: End-to-end RAG quality across simple, comparison, aggregation, multi-hop, set, post-processing, and false-premise questions, spanning finance, sports, music, movies, and open domains with varying degrees of real-time freshness.
Why it matters: CRAG's mock APIs and dynamic (real-time to static) question mix make it far closer to a production RAG assistant's actual query distribution than a static, purely-Wikipedia QA set.
Known limitations: Even the best-performing industry systems tested in the paper only reached 51% truthfulness with a 16-25% hallucination rate, showing the benchmark is far from saturated and results across different systems aren't always directly comparable given differing retrieval access.
What CRAG Measures
CRAG's 4,409 question-answer pairs split into 2,425 web-sourced and 1,984 knowledge-graph-sourced questions, covering eight question types (simple, simple-with-condition, set, comparison, aggregation, multi-hop, post-processing-heavy, and false-premise) across five domains. Each question is also tagged by "dynamism" — from real-time (answer changes by the second, e.g., stock prices) to static (never changes, e.g., birth dates) — deliberately stress-testing whether a system knows when its knowledge might be stale.
Three tasks build on each other: Task 1 provides 5 candidate web pages per question (answer generation only), Task 2 adds mock knowledge-graph APIs, and Task 3 provides 50 web pages (testing search ranking under noise). Scoring distinguishes perfect, acceptable, missing, and incorrect answers, explicitly penalizing hallucinated (incorrect) answers more than an honest "I don't know" — a design choice meant to discourage confidently wrong answers in production assistants.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Reasoning / retrieval-augmented factual QA |
| Metric | Truthfulness (accuracy − hallucination penalty, human- or auto-evaluated) |
| Number of tasks | 4,409 QA pairs across 5 domains, 8 question types |
| Saturation | Low — best industry RAG systems tested reached only 51% truthfulness |
| Created by | Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, et al. (Meta Reality Labs / FAIR, with HKUST collaborators) |
| Source paper | Yang et al. 2024 (NeurIPS 2024 D&B Track) |
| GitHub | facebookresearch/CRAG |
How CRAG Is Scored
Each answer is labeled perfect (+1), acceptable (+0.5), missing (0), or incorrect (−1); truthfulness is the average of these scores across the evaluation set. Auto-evaluation (using an LLM judge, validated at ~95-99% F1 against human labels) merges perfect/acceptable into "accurate" and reports accuracy, hallucination rate, missing rate, and the resulting truthfulness score together, since truthfulness alone can mask a high-hallucination, high-accuracy system.
State-of-the-Art Results
| Rank | Model | Accuracy (Task 3, end-to-end) | Source | Date |
|---|---|---|---|---|
| 1 | GPT-4 Turbo | 43.6% | Yang et al. 2024 | 2024-06 |
| 2 | Llama 3 70B Instruct | 40.6% | Yang et al. 2024 | 2024-06 |
| 3 | Mixtral-8x7B-Instruct-v0.1 | 33.5% | Yang et al. 2024 | 2024-06 |
| 4 | Llama 2 70B Chat | 31.9% | Yang et al. 2024 | 2024-06 |
Scores are Task 3 (end-to-end RAG with 50 candidate web pages + mock KG access) accuracy from the paper's straightforward-RAG-solution baselines, auto-evaluated. Truthfulness (which additionally penalizes hallucination) is lower for all models — see the paper for the full breakdown.
CRAG on Benchgen
Benchgen has synced the paper's own reported GPT-4 Turbo baseline score — see the live results table below. Run CRAG against your own model to add a verified entry.
CRAG vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| CRAG | RAG with mock web/KG APIs, hallucination-penalized | 4,409 | low |
| FRAMES | Multi-hop Wikipedia reasoning for RAG | 824 | low |
| PopQA | Single-hop entity factual recall | 14,000 | medium |
CRAG is distinguished by its dynamic-question design (real-time to static) and its explicit hallucination-penalized scoring, closer to a production RAG assistant's real query mix than FRAMES's purely Wikipedia-sourced multi-hop questions or PopQA's single-hop factual recall.
Run CRAG on Your Model
Benchgen lets teams evaluate their own RAG pipeline against CRAG's three escalating retrieval-access tasks, tracking accuracy, hallucination rate, and truthfulness together rather than a single number that can hide a high-hallucination system.