BrowseComp

BrowseComp

Quick answer: BrowseComp is OpenAI's benchmark for browsing agents (Wei et al., April 2025), comprising 1,266 questions that require persistent multi-hop web research to locate hard-to-find, entangled facts — questions that GPT-4o and humans both struggle to answer within a reasonable time. The "w/ context management" variant adds scaffolding for long-context browsing sessions. GPT-5.6 Sol leads the Inkling comparison set at 90.8%.

At a Glance

What it tests: A browsing agent's ability to find obscure, non-trivial facts that require navigating tens or hundreds of websites, reasoning about retrieved content, and reformulating searches — not just recalling memorised knowledge.

Why it matters: SimpleQA and TriviaQA saturated quickly as models gained web access. BrowseComp resets the difficulty bar: questions were designed so that GPT-4o (with and without browsing), o1, and an early deep research model all failed at creation time. Only persistent, creative, multi-hop browsing agents can score meaningfully. It is the standard benchmark for agentic web research capability as of 2025–2026.

Known limitations: Questions are deliberately inverted (start from the answer, build the question), which may not reflect natural user query distributions. Short-answer grading ignores the quality of long-form synthesis. Models trained specifically on BrowseComp-style tasks (like OpenAI's Deep Research) have a significant advantage.

What BrowseComp Measures

BrowseComp's 1,266 questions were created by human trainers who started from a known fact, identified several hard-to-search characteristics, and constructed an "inverted" question where the answer is easy to verify but hard to find. Each question was rejected if any of GPT-4o (w/ and w/o browsing), o1, or an early deep research model could solve it. Questions were also filtered to ensure no answer appeared on the first page of any of five common search queries.

Human trainers themselves solved only 29.2% of questions (after up to 2 hours of searching), confirming extreme difficulty. Topics span history, science, culture, entertainment, sports, and more, distributed according to trainer interests for quality.

The "w/ context management" variant evaluated in the Inkling table adds a context management scaffold that allows the agent to compress and summarise accumulated browsing history, enabling better long-horizon research over many pages without hitting context limits. Scores with this scaffold are higher than vanilla BrowseComp.

Benchmark Specifications

How BrowseComp Is Scored

A model (with web browsing enabled) is given each question and must return a short answer. Grading is performed by an LLM judge that compares the predicted answer to the reference answer and returns correct/incorrect/not-attempted. Scores are reported as % correct.

Performance scales with test-time compute: more browsing effort and aggregation strategies (majority vote, best-of-N) improve scores significantly. The Inkling table reports single-attempt scores with context management scaffolding at effort=0.99.

State-of-the-Art Results

Scores from Inkling model card (Thinking Machines Lab, July 2026), BrowseComp w/ context management, effort=0.99. Nemotron 3 Ultra and GLM 5.2 not reported.

BrowseComp vs Related Benchmarks

Last updated 2026-07-16.