# CRAG ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Truthfulness-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-4%2C409-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Jun%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2406.04744) [![GitHub](https://img.shields.io/badge/GitHub-facebookresearch%2FCRAG-181717?logo=github&logoColor=white)](https://github.com/facebookresearch/CRAG) > **Quick answer:** CRAG (Comprehensive RAG Benchmark) is a Meta-created factual question-answering benchmark of 4,409 QA pairs across five domains, paired with mock web-search and knowledge-graph APIs to simulate realistic retrieval-augmented generation, and scored with a truthfulness metric that penalizes hallucinated answers more heavily than missing ones. It laid the groundwork for the 2024 KDD Cup challenge. ## At a Glance **What it tests:** End-to-end RAG quality across simple, comparison, aggregation, multi-hop, set, post-processing, and false-premise questions, spanning finance, sports, music, movies, and open domains with varying degrees of real-time freshness. **Why it matters:** CRAG's mock APIs and dynamic (real-time to static) question mix make it far closer to a production RAG assistant's actual query distribution than a static, purely-Wikipedia QA set. **Known limitations:** Even the best-performing industry systems tested in the paper only reached 51% truthfulness with a 16-25% hallucination rate, showing the benchmark is far from saturated and results across different systems aren't always directly comparable given differing retrieval access. ## What CRAG Measures CRAG's 4,409 question-answer pairs split into 2,425 web-sourced and 1,984 knowledge-graph-sourced questions, covering eight question types (simple, simple-with-condition, set, comparison, aggregation, multi-hop, post-processing-heavy, and false-premise) across five domains. Each question is also tagged by "dynamism" — from real-time (answer changes by the second, e.g., stock prices) to static (never changes, e.g., birth dates) — deliberately stress-testing whether a system knows when its knowledge might be stale. Three tasks build on each other: Task 1 provides 5 candidate web pages per question (answer generation only), Task 2 adds mock knowledge-graph APIs, and Task 3 provides 50 web pages (testing search ranking under noise). Scoring distinguishes perfect, acceptable, missing, and incorrect answers, explicitly penalizing hallucinated (incorrect) answers more than an honest "I don't know" — a design choice meant to discourage confidently wrong answers in production assistants. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning / retrieval-augmented factual QA | | Metric | Truthfulness (accuracy − hallucination penalty, human- or auto-evaluated) | | Number of tasks | 4,409 QA pairs across 5 domains, 8 question types | | Saturation | Low — best industry RAG systems tested reached only 51% truthfulness | | Created by | Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, et al. (Meta Reality Labs / FAIR, with HKUST collaborators) | | Source paper | [Yang et al. 2024](https://arxiv.org/abs/2406.04744) (NeurIPS 2024 D&B Track) | | GitHub | [facebookresearch/CRAG](https://github.com/facebookresearch/CRAG) | ## How CRAG Is Scored Each answer is labeled perfect (+1), acceptable (+0.5), missing (0), or incorrect (−1); truthfulness is the average of these scores across the evaluation set. Auto-evaluation (using an LLM judge, validated at ~95-99% F1 against human labels) merges perfect/acceptable into "accurate" and reports accuracy, hallucination rate, missing rate, and the resulting truthfulness score together, since truthfulness alone can mask a high-hallucination, high-accuracy system. ## State-of-the-Art Results | Rank | Model | Accuracy (Task 3, end-to-end) | Source | Date | |---|---|---|---|---| | 1 | [GPT-4 Turbo](/models/openai/gpt-4-turbo) | 43.6% | [Yang et al. 2024](https://arxiv.org/abs/2406.04744) | 2024-06 | | 2 | Llama 3 70B Instruct | 40.6% | [Yang et al. 2024](https://arxiv.org/abs/2406.04744) | 2024-06 | | 3 | Mixtral-8x7B-Instruct-v0.1 | 33.5% | [Yang et al. 2024](https://arxiv.org/abs/2406.04744) | 2024-06 | | 4 | Llama 2 70B Chat | 31.9% | [Yang et al. 2024](https://arxiv.org/abs/2406.04744) | 2024-06 | *Scores are Task 3 (end-to-end RAG with 50 candidate web pages + mock KG access) accuracy from the paper's straightforward-RAG-solution baselines, auto-evaluated. Truthfulness (which additionally penalizes hallucination) is lower for all models — see the paper for the full breakdown.* ## CRAG on Benchgen Benchgen has synced the paper's own reported GPT-4 Turbo baseline score — see the live results table below. [Run CRAG against your own model](/benchmarks/xiao-yang-kai-sun-hao-xin-yushi-sun-et-al-meta/crag) to add a verified entry. ## CRAG vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **CRAG** | RAG with mock web/KG APIs, hallucination-penalized | 4,409 | low | | [FRAMES](/benchmarks/satyapriya-krishna-et-al-google-deepmind-harvard-university/frames) | Multi-hop Wikipedia reasoning for RAG | 824 | low | | [PopQA](/benchmarks/alex-mallen-et-al-university-of-washington-allen-institute-for-ai/popqa) | Single-hop entity factual recall | 14,000 | medium | CRAG is distinguished by its dynamic-question design (real-time to static) and its explicit hallucination-penalized scoring, closer to a production RAG assistant's real query mix than FRAMES's purely Wikipedia-sourced multi-hop questions or PopQA's single-hop factual recall. ## Run CRAG on Your Model Benchgen lets teams evaluate their own RAG pipeline against CRAG's three escalating retrieval-access tasks, tracking accuracy, hallucination rate, and truthfulness together rather than a single number that can hide a high-hallucination system. ## Frequently Asked Questions

What is CRAG?

CRAG (Comprehensive RAG Benchmark) is a Meta-created factual QA benchmark of 4,409 questions with mock web-search and knowledge-graph APIs, scored on a truthfulness metric that penalizes hallucination more than missing answers.

What does a good CRAG score look like?

The paper's own straightforward LLM-only and RAG baselines topped out around 33-44% accuracy, and even the best industry systems tested reached only 51% truthfulness — so scores in that range or higher represent strong, though still far from saturated, performance.

Who created CRAG?

CRAG was created by Xiao Yang, Kai Sun, and collaborators at Meta (Reality Labs and FAIR), with HKUST co-authors, published at NeurIPS 2024 and used as the basis for the KDD Cup 2024 challenge.

CRAG — Results

CRAG — Evaluations