MCP-Atlas

MCP-Atlas

Quick answer: MCP-Atlas is a large-scale benchmark by Scale AI that measures LLM agent tool-use competency against 36 production MCP (Model Context Protocol) servers. It contains 1,000 expert-written, multi-step tasks spanning 220 tools, scored using a claim-level rubric rather than exact-match. Prompts deliberately don't specify which tools to use, requiring agents to discover and compose the right tools autonomously. Claude Fable 5 leads the Inkling comparison set at 83.3%.

At a Glance

What it tests: An LLM agent's ability to identify relevant tools from a large toolspace, compose multi-step workflows across multiple MCP servers, and produce answers that satisfy atomic factual claims grounded in tool outputs.

Why it matters: Most tool-use benchmarks use mock servers or simple single-tool tasks. MCP-Atlas uses real production MCP servers, forcing models to deal with realistic tool schemas, server discovery, and multi-hop orchestration — the actual conditions of deployed agents. Its claim-level scoring also separates cognitive failures (wrong synthesis, premature stopping) from tool-call failures, providing actionable diagnostics.

Known limitations: The 500-task public split is freely available; a 500-task private split is held back to preserve leaderboard integrity, so local reproduction covers only half the benchmark. Scores depend on tool availability and reproducibility of external MCP servers.

What MCP-Atlas Measures

MCP-Atlas evaluates agents across three axes that prior MCP evaluations missed:

1. Realistic multi-step workflows: Tasks require cross-server orchestration — combining results from multiple tools across multiple MCP servers in sequence.
2. Breadth across authentic MCP servers: 36 real production MCP servers (not mocks) covering domains including search, calendar, code execution, databases, and more.
3. Structured claim-level scoring: Each task's expected answer is decomposed into atomic factual claims. An agent's response is scored by the fraction of claims it satisfies, with a default threshold of 0.75 (75% of claims must be satisfied for the task to count as passed). This allows alternate valid tool-call paths to receive credit.

The benchmark also includes an 11-category diagnostic taxonomy separating:

• Cognitive failures: incorrect synthesis, premature stopping, parsing errors (~63.3% of failures)
• Tool-call failures: wrong tool selection, parameter errors

Benchmark Specifications

How MCP-Atlas Is Scored

Each task has a natural-language prompt and a rubric of atomic factual claims. The agent executes tool calls and produces a final answer. An automated claim evaluator checks the answer against each claim. A task is "passed" if the fraction of satisfied claims ≥ 0.75. The reported score is the percentage of tasks passed at this threshold. This design allows valid alternative tool-call trajectories — agents that arrive at the correct answer through different tool combinations — to receive full credit.

State-of-the-Art Results

Scores from Inkling model card (Thinking Machines Lab, July 2026). All 9 comparison models reported.

MCP-Atlas vs Related Benchmarks

Last updated 2026-07-16.