# LongFact Concepts ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-F1%40K-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-%E2%89%881%2C140-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-medium-d29922) ![Created](https://img.shields.io/badge/Created-Mar%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2403.18802) [![GitHub](https://img.shields.io/badge/GitHub-long--form--factuality-181717?logo=github&logoColor=white)](https://github.com/google-deepmind/long-form-factuality) > **Quick answer:** LongFact-Concepts is the concept-explanation subset of Google DeepMind's LongFact prompt set — thousands of fact-seeking prompts spanning 38 topics designed to elicit long-form answers — scored automatically using the Search-Augmented Factuality Evaluator (SAFE), which was shown to agree with crowdsourced human raters 72% of the time at over 20x lower cost. ## At a Glance **What it tests:** Whether a model's long, open-ended explanations of concepts (as opposed to specific named entities) hold up fact-by-fact under an automated search-verified fact-checking process. **Why it matters:** Most factuality benchmarks test short-answer QA; LongFact instead evaluates whether long, discursive model output — the kind users actually read in real usage — remains factually accurate throughout, not just in a single terminal answer. **Known limitations:** SAFE's automated grading relies on Google Search as its evidence source and an LLM to adjudicate each individual fact, so its accuracy is bounded by both search-result quality and the judge model's own reasoning ability. ## What LongFact Concepts Measures LongFact is built by using GPT-4 to generate a large prompt set — 2,280 fact-seeking prompts in total — spanning 38 topics, explicitly designed to require long-form (multi-paragraph) responses rather than single-fact answers. The full prompt set splits into two subsets by question type: **LongFact-Concepts**, covering prompts that ask a model to explain or describe a concept (e.g., "what is quantum entanglement and how does it work?"), and LongFact-Objects, covering prompts about specific named entities. Because manually fact-checking long-form output at scale is prohibitively expensive, the paper introduces the **Search-Augmented Factuality Evaluator (SAFE)**: an LLM-agent pipeline that decomposes a long-form response into individual atomic facts, issues Google Search queries for each fact, and determines whether the fact is supported by the search results. SAFE agreed with crowdsourced human annotators 72% of the time on a set of ~16,000 individual facts, and won 76% of a sampled set of disagreement cases when independently re-checked — all while being more than 20x cheaper than human annotation. The paper also introduces F1@K, extending the standard F1 score to long-form settings by balancing the percentage of supported facts (precision) against a length-normalized measure of how many facts were provided relative to a user's preferred response length (recall). ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning / long-form factuality (concept-explanation prompts) | | Metric | F1@K — SAFE-scored precision/recall over decomposed atomic facts | | Number of tasks | ~1,140 (concept-explanation half of the 2,280-prompt LongFact set) | | Saturation | Medium — larger models generally score better, but no model is fully saturated | | Created by | Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, Quoc V. Le (Google DeepMind) | | Source paper | [Wei et al. 2024](https://arxiv.org/abs/2403.18802) (NeurIPS 2024) | | GitHub | [google-deepmind/long-form-factuality](https://github.com/google-deepmind/long-form-factuality) | ## How LongFact Concepts Is Scored SAFE breaks each model response into individual atomic facts, then verifies each fact against Google Search results via a multi-step reasoning process, classifying it as supported, unsupported, or irrelevant. The F1@K metric combines the resulting precision (share of facts that are actually supported) with a recall term capped at a hyperparameter K representing a user's preferred response length, so a model cannot game the score by simply omitting risky claims or padding with excessive length. ## State-of-the-Art Results > The original paper benchmarks 13 language models across four families (Gemini, GPT, Claude, PaLM-2), finding that larger models within each family generally achieve better long-form factuality. See the [official GitHub repository](https://github.com/google-deepmind/long-form-factuality) for the full per-model breakdown. ## LongFact Concepts on Benchgen > No Benchgen results yet — [be the first to run LongFact Concepts](/benchmarks/jerry-wei-chengrun-yang-xinying-song-yifeng-lu-nathan-hu-et-al-google-deepmind/longfact-concepts). ## LongFact Concepts vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **LongFact Concepts** | Long-form factuality on concept-explanation prompts | ~1,140 | medium | | [LongFact Objects](/benchmarks/jerry-wei-chengrun-yang-xinying-song-yifeng-lu-nathan-hu-et-al-google-deepmind/longfact-objects) | Long-form factuality on named-entity prompts | ~1,140 | medium | | [PopQA](/benchmarks/alex-mallen-et-al-university-of-washington-allen-institute-for-ai/popqa) | Short-answer factual QA on long-tail entities | — | low | LongFact-Concepts is distinguished from short-answer factuality benchmarks like PopQA by testing long, multi-paragraph responses fact-checked at the granularity of individual atomic claims, rather than a single terminal answer string. ## Run LongFact Concepts on Your Model Benchgen lets teams evaluate their own model's long-form factuality on concept-explanation prompts using the SAFE methodology, tracking fact-level precision and recall trends across releases. ## Frequently Asked Questions

What is LongFact-Concepts?

LongFact-Concepts is the concept-explanation subset of Google DeepMind's LongFact prompt set, testing long-form factuality using the automated SAFE evaluator.

What does a good LongFact Concepts score look like?

The original paper found larger models generally achieve better long-form factuality within each model family, so a high F1@K score sustained across many long, multi-fact responses represents strong performance.

Who created LongFact?

LongFact was created by Jerry Wei, Chengrun Yang, and collaborators at Google DeepMind, published at NeurIPS 2024.

LongFact Concepts — Phases

LongFact Concepts