LongFact Concepts

LongFact Concepts

Quick answer: LongFact-Concepts is the concept-explanation subset of Google DeepMind's LongFact prompt set — thousands of fact-seeking prompts spanning 38 topics designed to elicit long-form answers — scored automatically using the Search-Augmented Factuality Evaluator (SAFE), which was shown to agree with crowdsourced human raters 72% of the time at over 20x lower cost.

At a Glance

What it tests: Whether a model's long, open-ended explanations of concepts (as opposed to specific named entities) hold up fact-by-fact under an automated search-verified fact-checking process.

Why it matters: Most factuality benchmarks test short-answer QA; LongFact instead evaluates whether long, discursive model output — the kind users actually read in real usage — remains factually accurate throughout, not just in a single terminal answer.

Known limitations: SAFE's automated grading relies on Google Search as its evidence source and an LLM to adjudicate each individual fact, so its accuracy is bounded by both search-result quality and the judge model's own reasoning ability.

What LongFact Concepts Measures

LongFact is built by using GPT-4 to generate a large prompt set — 2,280 fact-seeking prompts in total — spanning 38 topics, explicitly designed to require long-form (multi-paragraph) responses rather than single-fact answers. The full prompt set splits into two subsets by question type: LongFact-Concepts, covering prompts that ask a model to explain or describe a concept (e.g., "what is quantum entanglement and how does it work?"), and LongFact-Objects, covering prompts about specific named entities.

Because manually fact-checking long-form output at scale is prohibitively expensive, the paper introduces the Search-Augmented Factuality Evaluator (SAFE): an LLM-agent pipeline that decomposes a long-form response into individual atomic facts, issues Google Search queries for each fact, and determines whether the fact is supported by the search results. SAFE agreed with crowdsourced human annotators 72% of the time on a set of ~16,000 individual facts, and won 76% of a sampled set of disagreement cases when independently re-checked — all while being more than 20x cheaper than human annotation. The paper also introduces F1@K, extending the standard F1 score to long-form settings by balancing the percentage of supported facts (precision) against a length-normalized measure of how many facts were provided relative to a user's preferred response length (recall).

Benchmark Specifications

How LongFact Concepts Is Scored

SAFE breaks each model response into individual atomic facts, then verifies each fact against Google Search results via a multi-step reasoning process, classifying it as supported, unsupported, or irrelevant. The F1@K metric combines the resulting precision (share of facts that are actually supported) with a recall term capped at a hyperparameter K representing a user's preferred response length, so a model cannot game the score by simply omitting risky claims or padding with excessive length.

State-of-the-Art Results

The original paper benchmarks 13 language models across four families (Gemini, GPT, Claude, PaLM-2), finding that larger models within each family generally achieve better long-form factuality. See the official GitHub repository for the full per-model breakdown.

LongFact Concepts on Benchgen

No Benchgen results yet — be the first to run LongFact Concepts.

LongFact Concepts vs Other Benchmarks

LongFact-Concepts is distinguished from short-answer factuality benchmarks like PopQA by testing long, multi-paragraph responses fact-checked at the granularity of individual atomic claims, rather than a single terminal answer string.

Run LongFact Concepts on Your Model

Benchgen lets teams evaluate their own model's long-form factuality on concept-explanation prompts using the SAFE methodology, tracking fact-level precision and recall trends across releases.

Frequently Asked Questions