# ACEBench ![Category](https://img.shields.io/badge/Category-Tool%20Usage%20%2F%20Agent-1f6feb) ![Metric](https://img.shields.io/badge/Metric-Score%20(0--1)-2da44e) ![Tasks](https://img.shields.io/badge/Tasks-4%2C538%20APIs-8957e5) ![Saturation](https://img.shields.io/badge/Saturation-Low-d29922) ![Created](https://img.shields.io/badge/Created-Jan%202025-555) [![Paper](https://img.shields.io/badge/Paper-arXiv%202501.12851-b31b1b)](https://arxiv.org/abs/2501.12851) [![GitHub](https://img.shields.io/badge/GitHub-ACEBench%2FACEBench-181717?logo=github&logoColor=white)](https://github.com/ACEBench/ACEBench) > **Quick answer:** ACEBench (Chen et al., January 2025) is a comprehensive tool-usage benchmark that evaluates LLMs across 4,538 APIs spanning 8 major domains and 68 sub-domains, using three evaluation types — Normal, Special, and Agent — to cover everything from basic function-calling to ambiguous instructions and multi-turn agentic dialogues. It removes reliance on live API execution or LLM-as-judge by using structured, rule-based evaluation, making results reproducible and low-overhead. ## At a Glance **What it tests:** An LLM's ability to correctly select, format, and invoke tools across basic, ambiguous, and multi-turn agentic scenarios covering domains including technology, finance, healthcare, entertainment, and environment. **Why it matters:** Prior tool-use benchmarks tested narrow scenarios and relied on either real API execution (expensive and non-deterministic) or LLM judges (high cost, potential bias). ACEBench provides structured ground-truth evaluation across a broad API surface, making it a practical standard for assessing production-grade tool-calling reliability. **Known limitations:** The dataset is not yet publicly released, limiting independent reproduction. With only 2 models tracked on llm-stats.com as of mid-2026, the leaderboard is sparse and saturation characteristics remain unclear. ## What ACEBench Measures ACEBench addresses three widely observed gaps in tool-usage evaluation: limited scenario coverage, narrow evaluation dimensions, and expensive evaluation pipelines. The benchmark organizes its 4,538 API entries across 8 major domains (technology, finance, entertainment, society, health, culture, environment, and general) and 68 sub-domains, supporting evaluation in both English and Chinese. The three evaluation types reflect a progression of real-world difficulty. **Normal** tasks test baseline tool-calling competence — given a clear instruction and a set of available APIs, can the model select the right tool and supply the correct arguments? **Special** tasks introduce deliberate ambiguity or incompleteness in the instruction, testing whether the model can handle underspecified requests gracefully rather than hallucinating a call or failing silently. **Agent** tasks simulate full multi-turn dialogues, evaluating whether the model can coordinate tool calls across multiple turns as a participant in a multi-agent workflow. Evaluation uses structured rule-based matching against ground-truth API calls, avoiding the latency and non-determinism of live API execution and the cost and potential bias of LLM-as-judge scoring. Scores are reported on a 0–1 scale aggregated across all three evaluation types. ## Benchmark Specifications | Field | Value | |---|---| | Task category | Tool usage / Agent | | Metric | Score (0–1) | | Number of APIs | 4,538 across 8 domains | | Evaluation types | Normal, Special, Agent | | Languages | English, Chinese | | Saturation | Low | | Created by | Chen et al. (Huawei Noah's Ark Lab) | | Source paper | [Chen et al. 2025](https://arxiv.org/abs/2501.12851) | | GitHub | [ACEBench/ACEBench](https://github.com/ACEBench/ACEBench) | | Dataset | Not yet public | ## How ACEBench Is Scored Each test case maps an instruction and available API list to a ground-truth API call (tool name + arguments). The model's output is matched against this ground-truth using rule-based structural comparison — checking whether the correct tool was selected and whether all required arguments are correctly populated. Scores are averaged across all cases in the three evaluation subtypes and reported as a single aggregate value between 0 and 1, where 1.0 represents perfect tool selection and argument filling on every case. ## ACEBench on Benchgen > No Benchgen results yet — [be the first to run ACEBench](/benchmarks/huawei/acebench). ## ACEBench vs Other Benchmarks | Benchmark | What it tests | APIs / Tasks | Saturation | |---|---|---|---| | **ACEBench** | Tool usage: Normal, Special, Agent | 4,538 APIs | Low | | [BFCL v3](/benchmarks/berkeley/bfcl-v3) | Function calling accuracy | ~2,000 tasks | Medium | | [Tau3 Banking](/benchmarks/sierra-research/3-banking) | Agentic task completion in banking | ~200 tasks | Low | | [MCP Atlas](/benchmarks/scale-ai/mcp-atlas) | MCP tool-use in agentic workflows | ~500 tasks | Low | ACEBench's three-tier design (Normal → Special → Agent) and bilingual coverage make it more comprehensive than single-scenario function-calling benchmarks; use BFCL for a focused API-matching signal and ACEBench when you need coverage of ambiguous instructions and multi-turn agentic behaviour. ## Run ACEBench on Your Model Rather than relying on vendor-reported one-time ACEBench numbers, Benchgen lets you run the benchmark against your own model versions, track scores across releases, and detect regressions automatically — so you know whether a fine-tune or prompt change improved or hurt tool-calling reliability before it ships. ## Frequently Asked Questions

What is ACEBench?

ACEBench is a benchmark for evaluating LLM tool usage across 4,538 APIs spanning 8 major domains, using three evaluation types: Normal (basic tool usage), Special (ambiguous or incomplete instructions), and Agent (multi-turn agentic interactions). It was introduced by Chen et al. at Huawei Noah's Ark Lab in January 2025.

What does a good score look like on ACEBench?

Scores are reported on a 0–1 scale. As of mid-2026, the highest publicly reported score is 0.765 (Kimi K2 Instruct), suggesting the benchmark remains far from saturation and leaves substantial room for improvement across all evaluation types.

Who created ACEBench?

ACEBench was created by Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, and colleagues at Huawei Noah's Ark Lab. The paper was submitted to arXiv on January 22, 2025 (arXiv:2501.12851).

Rank	Model	Score
1	gpt-4o	0.896
2	qwen2-5-coder-32b-instruct	0.853
3	qwen2-5-32b-instruct	0.799
4	qwen2-5-72b-instruct	0.793
5	kimi-k2-instruct-0905	0.765
6	kimi-k2-instruct	0.765
7	claude-3-5-sonnet	0.756
8	gemini-1-5-pro	0.728
9	qwen2-5-14b-instruct	0.64
10	llama-3-1-70b-instruct	0.629
11	qwen2-5-7b-instruct	0.578
12	qwen2-5-coder-7b-instruct	0.496
13	llama-3-1-8b-instruct	0.338
14	phi-3-5-mini-instruct	0.295

ACEBench — Results

ACEBench — Evaluations