ToolSandbox

ToolSandbox

Quick answer: ToolSandbox (Lu et al., Apple, Aug 2024) is a stateful, conversational benchmark for evaluating LLM tool use. Unlike prior tool-use benchmarks that test stateless, single-turn API calls, ToolSandbox includes stateful tool execution, implicit dependencies between tools, a built-in user simulator for on-policy conversational evaluation, and dynamic scoring across intermediate and final milestones.

At a Glance

What it tests: Whether an LLM agent can correctly use tools whose state persists and changes across a conversation, handle implicit dependencies between tool calls, and converse naturally with a simulated user rather than following a fixed, off-policy script.

Why it matters: Most real agentic tool use is stateful (e.g. booking a flight changes what seats are subsequently available) and conversational (users clarify and correct mid-task). ToolSandbox specifically targets the gap between benchmarks that test isolated, stateless API calls and the messier reality of production agents.

Known limitations: The paper reports a significant performance gap between open and proprietary models on ToolSandbox's harder categories (state dependency, canonicalization, insufficient information), meaning scores can vary widely by task subset.

What ToolSandbox Measures

ToolSandbox evaluates tool-use capability under conditions closer to real deployment than earlier tool-calling benchmarks: tools have persistent state (calling one tool can change what a later tool call should return or require), some tasks have implicit dependencies between tools that the model must infer rather than being told explicitly, and a built-in user simulator carries on an actual multi-turn conversation with the model under test (on-policy), rather than replaying a fixed transcript.

Scoring uses a dynamic evaluation strategy that checks for both intermediate milestones (did the agent make the right state-changing calls along the way) and the final outcome, over an arbitrary-length trajectory. The paper highlights three categories — State Dependency, Canonicalization, and Insufficient Information — as challenging even the strongest proprietary models at time of publication, and finds a substantial performance gap between open-source and proprietary systems.

Benchmark Specifications

How ToolSandbox Is Scored

Each task is evaluated across a trajectory of arbitrary length rather than a single tool call. The evaluator checks whether the agent hits the intermediate milestones needed along the way (e.g. correctly querying state-dependent tools in the right order) as well as the final task outcome, and aggregates these into a milestone completion rate. Because the user simulator is on-policy (it responds dynamically to what the model actually says), no two runs follow an identical script, which more closely mirrors production conversational agents than static, pre-scripted tool-use benchmarks.

ToolSandbox on Benchgen

No Benchgen results yet — be the first to run ToolSandbox.

ToolSandbox vs Other Benchmarks

ToolSandbox is the better pick for evaluating conversational, state-aware agent behavior; BFCL's AST-based scoring is more deterministic and reproducible for isolated function-call correctness.

FAQ

What is ToolSandbox? ToolSandbox is Apple's benchmark for stateful, conversational LLM tool use, combining a live user simulator with dynamic milestone-based scoring across arbitrary-length trajectories.

How is ToolSandbox scored? Scoring is based on milestone completion — whether the agent correctly hits intermediate state-changing checkpoints and the final task outcome — across a full conversational trajectory rather than a single tool call.

Who created ToolSandbox? Lu et al. at Apple, published on arXiv in August 2024 (arXiv:2408.04682).

Is ToolSandbox saturated? No — the original paper found a significant gap between open-source and proprietary models on its hardest categories (State Dependency, Canonicalization, Insufficient Information), indicating substantial headroom remains.

How does ToolSandbox differ from BFCL? BFCL evaluates function-calling correctness via deterministic AST matching against ground-truth calls; ToolSandbox additionally models persistent tool state and uses a live, on-policy user simulator instead of a fixed conversational script.

Benchmark definition paraphrased from Lu et al. 2024 (arXiv:2408.04682). Last updated 2026-08-04.