ACEBench

ACEBench

Quick answer: ACEBench (Chen et al., January 2025) is a comprehensive tool-usage benchmark that evaluates LLMs across 4,538 APIs spanning 8 major domains and 68 sub-domains, using three evaluation types — Normal, Special, and Agent — to cover everything from basic function-calling to ambiguous instructions and multi-turn agentic dialogues. It removes reliance on live API execution or LLM-as-judge by using structured, rule-based evaluation, making results reproducible and low-overhead.

At a Glance

What it tests: An LLM's ability to correctly select, format, and invoke tools across basic, ambiguous, and multi-turn agentic scenarios covering domains including technology, finance, healthcare, entertainment, and environment.

Why it matters: Prior tool-use benchmarks tested narrow scenarios and relied on either real API execution (expensive and non-deterministic) or LLM judges (high cost, potential bias). ACEBench provides structured ground-truth evaluation across a broad API surface, making it a practical standard for assessing production-grade tool-calling reliability.

Known limitations: The dataset is not yet publicly released, limiting independent reproduction. With only 2 models tracked on llm-stats.com as of mid-2026, the leaderboard is sparse and saturation characteristics remain unclear.

What ACEBench Measures

ACEBench addresses three widely observed gaps in tool-usage evaluation: limited scenario coverage, narrow evaluation dimensions, and expensive evaluation pipelines. The benchmark organizes its 4,538 API entries across 8 major domains (technology, finance, entertainment, society, health, culture, environment, and general) and 68 sub-domains, supporting evaluation in both English and Chinese.

The three evaluation types reflect a progression of real-world difficulty. Normal tasks test baseline tool-calling competence — given a clear instruction and a set of available APIs, can the model select the right tool and supply the correct arguments? Special tasks introduce deliberate ambiguity or incompleteness in the instruction, testing whether the model can handle underspecified requests gracefully rather than hallucinating a call or failing silently. Agent tasks simulate full multi-turn dialogues, evaluating whether the model can coordinate tool calls across multiple turns as a participant in a multi-agent workflow.

Evaluation uses structured rule-based matching against ground-truth API calls, avoiding the latency and non-determinism of live API execution and the cost and potential bias of LLM-as-judge scoring. Scores are reported on a 0–1 scale aggregated across all three evaluation types.

Benchmark Specifications

How ACEBench Is Scored

Each test case maps an instruction and available API list to a ground-truth API call (tool name + arguments). The model's output is matched against this ground-truth using rule-based structural comparison — checking whether the correct tool was selected and whether all required arguments are correctly populated. Scores are averaged across all cases in the three evaluation subtypes and reported as a single aggregate value between 0 and 1, where 1.0 represents perfect tool selection and argument filling on every case.

ACEBench on Benchgen

No Benchgen results yet — be the first to run ACEBench.

ACEBench vs Other Benchmarks

ACEBench's three-tier design (Normal → Special → Agent) and bilingual coverage make it more comprehensive than single-scenario function-calling benchmarks; use BFCL for a focused API-matching signal and ACEBench when you need coverage of ambiguous instructions and multi-turn agentic behaviour.

Run ACEBench on Your Model

Rather than relying on vendor-reported one-time ACEBench numbers, Benchgen lets you run the benchmark against your own model versions, track scores across releases, and detect regressions automatically — so you know whether a fine-tune or prompt change improved or hurt tool-calling reliability before it ships.

Frequently Asked Questions