LongFact Objects
1 phaseActive
Google DeepMind's long-form factuality prompt set (named-entity subset) spanning 38 topics, auto-graded by the SAFE search-augmented evaluator (NeurIPS 2024).
Overview
LongFact Objects
Quick answer: LongFact-Objects is the named-entity subset of Google DeepMind's LongFact prompt set — thousands of fact-seeking prompts spanning 38 topics that ask about specific people, places, or things — scored automatically using the Search-Augmented Factuality Evaluator (SAFE), which agreed with human raters 72% of the time at over 20x lower cost.
At a Glance
What it tests: Whether a model's long, open-ended answers about specific named entities (people, organizations, places, works) hold up fact-by-fact under automated search-verified checking, as opposed to abstract concept explanations.
Why it matters: Named-entity factual errors (wrong dates, wrong affiliations, fabricated achievements) are among the most common and reputationally damaging hallucination failure modes in production LLM deployments, making this subset especially relevant to real-world factuality risk.
Known limitations: Like LongFact-Concepts, SAFE's grading quality is bounded by Google Search result quality and the judge LLM's own reasoning, and entities with sparse or contested web coverage may be harder to verify reliably.
What LongFact Objects Measures
LongFact-Objects is one of the two subsets of the full LongFact prompt set (2,280 prompts total, generated using GPT-4, spanning 38 topics). While LongFact-Concepts asks a model to explain abstract concepts, LongFact-Objects instead asks about specific named entities — for example, biographical or historical facts about a particular person, organization, or place — requiring a long-form response that a grader must verify claim-by-claim.
Grading uses the same Search-Augmented Factuality Evaluator (SAFE) introduced alongside LongFact: an LLM agent decomposes each long-form response into individual atomic facts and verifies each one against Google Search results through a multi-step reasoning process. On a set of roughly 16,000 individual facts, SAFE's verdicts agreed with crowdsourced human annotators 72% of the time, and on a sampled set of disagreement cases, SAFE's verdict was independently judged correct 76% of the time — all at more than 20x lower cost than human annotation. Scores are reported using F1@K, which balances the fraction of supported facts (precision) against a length-normalized recall term.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Reasoning / long-form factuality (named-entity prompts) |
| Metric | F1@K — SAFE-scored precision/recall over decomposed atomic facts |
| Number of tasks | ~1,140 (named-entity half of the 2,280-prompt LongFact set) |
| Saturation | Medium — larger models generally score better, but no model is fully saturated |
| Created by | Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, Quoc V. Le (Google DeepMind) |
| Source paper | Wei et al. 2024 (NeurIPS 2024) |
| GitHub | google-deepmind/long-form-factuality |
How LongFact Objects Is Scored
Each response is decomposed by SAFE into individual atomic factual claims about the named entity in question, and each claim is checked against Google Search evidence and classified as supported, unsupported, or irrelevant. The headline F1@K metric combines precision (share of claims actually supported) with a length-capped recall term, discouraging both hallucinated claims and evasive under-length responses.
State-of-the-Art Results
The original paper benchmarks 13 language models across four families (Gemini, GPT, Claude, PaLM-2), finding that larger models within each family generally achieve better long-form factuality. See the official GitHub repository for the full per-model breakdown.
LongFact Objects on Benchgen
No Benchgen results yet — be the first to run LongFact Objects.
LongFact Objects vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| LongFact Objects | Long-form factuality on named-entity prompts | ~1,140 | medium |
| LongFact Concepts | Long-form factuality on concept-explanation prompts | ~1,140 | medium |
| PopQA | Short-answer factual QA on long-tail entities | — | low |
LongFact-Objects complements LongFact-Concepts by focusing specifically on named-entity factual recall within long-form responses, the failure mode most commonly associated with production hallucination incidents.
Run LongFact Objects on Your Model
Benchgen lets teams evaluate their own model's long-form factuality on named-entity prompts using the SAFE methodology, tracking fact-level precision and recall trends across releases.