FRAMES

FRAMES

Quick answer: FRAMES (Factuality, Retrieval, And reasoning MEasurement Set) is a Google DeepMind/Harvard benchmark of 824 challenging multi-hop questions that unifies factuality, retrieval, and reasoning evaluation for retrieval-augmented generation (RAG) systems in one end-to-end test, rather than assessing each capability in isolation.

At a Glance

What it tests: Whether a RAG pipeline can retrieve and correctly reason across 2-15 Wikipedia articles to answer a single multi-hop question involving numerical, temporal, tabular, or multi-constraint reasoning.

Why it matters: Most RAG benchmarks test retrieval, factuality, or reasoning separately; FRAMES demonstrates that even models with perfect retrieval ("oracle" context) still make significant reasoning errors, showing that retrieval alone doesn't solve the problem.

Known limitations: All source documents are drawn from Wikipedia, which narrows real-world domain coverage and introduces some pretraining-contamination risk despite deliberate question design to require extra reasoning steps beyond simple lookup.

What FRAMES Measures

FRAMES questions are constructed by human annotators to require combining information from multiple Wikipedia articles — roughly 36% of questions need two articles, 35% need three, and the rest need four or more, up to 15. Each question is also labeled with one or more of five reasoning types: numerical reasoning, tabular reasoning, multiple constraints, temporal reasoning, and post-processing (e.g., converting a computed year to Roman numerals).

The paper's core finding is stark: a strong model (Gemini-Pro-1.5) answers only 40.8% of questions correctly with no retrieval, improves to 47.4% with BM25-retrieved context, and reaches 66% with a proposed multi-step retrieval-and-reasoning pipeline — still well short of the 72.9% "oracle" ceiling reached when all correct source articles are handed to the model directly. The gap between oracle and even the best iterative pipeline shows models still make reasoning errors — concentrated in numerical, tabular, and post-processing question types — even once retrieval is solved.

Benchmark Specifications

How FRAMES Is Scored

An LLM autorater (validated at 0.96 accuracy and 0.889 Cohen's Kappa against human judgment) determines whether a model's free-form answer matches the gold answer. Because FRAMES separates naive (no-retrieval), BM25-retrieved, multi-step, and oracle-context conditions, published results typically report accuracy under each retrieval condition rather than a single aggregate number.

State-of-the-Art Results

Scores shown are the paper's "Naive Prompt" (no-retrieval) condition — the paper's own reported lower bound. Multi-step retrieval and oracle-context conditions score substantially higher for all models; see the paper for the full breakdown.

FRAMES on Benchgen

Benchgen has synced the paper's own reported no-retrieval baseline scores for Gemini 1.5 Pro, Gemini 1.5 Flash, Gemma 2 27B, and Llama 3.2 3B Instruct — see the live results table below. Run FRAMES against your own model to add a verified entry.

FRAMES vs Other Benchmarks

FRAMES is distinguished from CRAG by its exclusive focus on multi-hop reasoning across Wikipedia articles (vs. CRAG's broader web/KG retrieval and dynamic real-time questions), and from PopQA by requiring genuine multi-document reasoning rather than single-fact lookup.

Run FRAMES on Your Model

Benchgen lets teams evaluate their own RAG pipeline or model against FRAMES's naive, retrieval-augmented, and oracle conditions, isolating whether performance gaps come from retrieval failures or genuine reasoning errors.

Frequently Asked Questions