# FRAMES ![Category](https://img.shields.io/badge/Category-Reasoning-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Accuracy-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-824-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-low-d29922) ![Created](https://img.shields.io/badge/Created-Sep%202024-555) [![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b)](https://arxiv.org/abs/2409.12941) [![Dataset](https://img.shields.io/badge/Dataset-HuggingFace-ff9d00?logo=huggingface&logoColor=white)](https://huggingface.co/datasets/google/frames-benchmark) > **Quick answer:** FRAMES (Factuality, Retrieval, And reasoning MEasurement Set) is a Google DeepMind/Harvard benchmark of 824 challenging multi-hop questions that unifies factuality, retrieval, and reasoning evaluation for retrieval-augmented generation (RAG) systems in one end-to-end test, rather than assessing each capability in isolation. ## At a Glance **What it tests:** Whether a RAG pipeline can retrieve and correctly reason across 2-15 Wikipedia articles to answer a single multi-hop question involving numerical, temporal, tabular, or multi-constraint reasoning. **Why it matters:** Most RAG benchmarks test retrieval, factuality, or reasoning separately; FRAMES demonstrates that even models with perfect retrieval ("oracle" context) still make significant reasoning errors, showing that retrieval alone doesn't solve the problem. **Known limitations:** All source documents are drawn from Wikipedia, which narrows real-world domain coverage and introduces some pretraining-contamination risk despite deliberate question design to require extra reasoning steps beyond simple lookup. ## What FRAMES Measures FRAMES questions are constructed by human annotators to require combining information from multiple Wikipedia articles — roughly 36% of questions need two articles, 35% need three, and the rest need four or more, up to 15. Each question is also labeled with one or more of five reasoning types: numerical reasoning, tabular reasoning, multiple constraints, temporal reasoning, and post-processing (e.g., converting a computed year to Roman numerals). The paper's core finding is stark: a strong model (Gemini-Pro-1.5) answers only 40.8% of questions correctly with no retrieval, improves to 47.4% with BM25-retrieved context, and reaches 66% with a proposed multi-step retrieval-and-reasoning pipeline — still well short of the 72.9% "oracle" ceiling reached when all correct source articles are handed to the model directly. The gap between oracle and even the best iterative pipeline shows models still make reasoning errors — concentrated in numerical, tabular, and post-processing question types — even once retrieval is solved. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Reasoning / unified RAG evaluation | | Metric | Accuracy (LLM-judged answer match, validated against human agreement) | | Number of tasks | 824 multi-hop questions | | Saturation | Low — even oracle-context performance tops out at ~73% | | Created by | Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, Manaal Faruqui (Google DeepMind / Harvard) | | Source paper | [Krishna et al. 2024](https://arxiv.org/abs/2409.12941) | | Dataset | [google/frames-benchmark](https://huggingface.co/datasets/google/frames-benchmark) | ## How FRAMES Is Scored An LLM autorater (validated at 0.96 accuracy and 0.889 Cohen's Kappa against human judgment) determines whether a model's free-form answer matches the gold answer. Because FRAMES separates naive (no-retrieval), BM25-retrieved, multi-step, and oracle-context conditions, published results typically report accuracy under each retrieval condition rather than a single aggregate number. ## State-of-the-Art Results | Rank | Model | Score (naive, no retrieval) | Source | Date | |---|---|---|---|---| | 1 | [Gemini 1.5 Pro](/models/google-deepmind/gemini-1-5-pro) | 40.8% | [Krishna et al. 2024](https://arxiv.org/abs/2409.12941) | 2024-09 | | 2 | [Gemma 2 27B](/models/google/gemma-2-27b) | 30.8% | [Krishna et al. 2024](https://arxiv.org/abs/2409.12941) | 2024-09 | | 3 | [Gemini 1.5 Flash](/models/google/gemini-1-5-flash) | 26.3% | [Krishna et al. 2024](https://arxiv.org/abs/2409.12941) | 2024-09 | | 4 | [Llama 3.2 3B Instruct](/models/meta/llama-3-2-3b-instruct) | 11.5% | [Krishna et al. 2024](https://arxiv.org/abs/2409.12941) | 2024-09 | *Scores shown are the paper's "Naive Prompt" (no-retrieval) condition — the paper's own reported lower bound. Multi-step retrieval and oracle-context conditions score substantially higher for all models; see the paper for the full breakdown.* ## FRAMES on Benchgen Benchgen has synced the paper's own reported no-retrieval baseline scores for Gemini 1.5 Pro, Gemini 1.5 Flash, Gemma 2 27B, and Llama 3.2 3B Instruct — see the live results table below. [Run FRAMES against your own model](/benchmarks/satyapriya-krishna-et-al-google-deepmind-harvard-university/frames) to add a verified entry. ## FRAMES vs Other Benchmarks | Benchmark | What it tests | Tasks | Saturation | |---|---|---|---| | **FRAMES** | Unified factuality + retrieval + reasoning for RAG | 824 | low | | [CRAG](/benchmarks/xiao-yang-kai-sun-hao-xin-yushi-sun-et-al-meta/crag) | RAG with mock APIs, hallucination-penalized scoring | 4,409 | low | | [PopQA](/benchmarks/alex-mallen-et-al-university-of-washington-allen-institute-for-ai/popqa) | Single-hop entity factual recall | 14,000 | medium | FRAMES is distinguished from CRAG by its exclusive focus on multi-hop reasoning across Wikipedia articles (vs. CRAG's broader web/KG retrieval and dynamic real-time questions), and from PopQA by requiring genuine multi-document reasoning rather than single-fact lookup. ## Run FRAMES on Your Model Benchgen lets teams evaluate their own RAG pipeline or model against FRAMES's naive, retrieval-augmented, and oracle conditions, isolating whether performance gaps come from retrieval failures or genuine reasoning errors. ## Frequently Asked Questions

What is FRAMES?

FRAMES is a Google DeepMind benchmark of 824 multi-hop questions that unifies factuality, retrieval, and reasoning evaluation for RAG systems in a single end-to-end test.

What does a good FRAMES score look like?

Even the strongest tested baseline (Gemini 1.5 Pro) only reached 40.8% accuracy with no retrieval and 66% with a multi-step retrieval pipeline — well below the 72.9% oracle ceiling — so scores above 50-60% under realistic retrieval conditions represent strong performance.

Who created FRAMES?

FRAMES was created by Satyapriya Krishna and collaborators at Google DeepMind and Harvard University, published in 2024.

Rank	Model	Score
1	gemini-1-5-pro	40.8
2	gemma-2-27b	30.8
3	gemini-1-5-flash	26.3
4	llama-3-2-3b-instruct	11.5

FRAMES — Results

FRAMES — Phases

FRAMES