API-Bank
1 phaseActive
Runnable tool-use benchmark with 73 API tools and 314 annotated dialogues (753 API calls), from Alibaba DAMO Academy, testing planning/retrieval/calling (EMNLP 2023).
Overview
API-Bank
Quick answer: API-Bank is a benchmark from Alibaba's DAMO Academy built around a runnable evaluation system of 73 API tools and 314 human-annotated tool-use dialogues (753 total API calls), testing how effectively LLMs can plan which API to use, retrieve the right one, and correctly call it.
At a Glance
What it tests: Three sequential tool-use skills — planning which API(s) a request requires, retrieving the correct API from a pool of candidates, and correctly calling it with valid parameters — evaluated against a genuinely runnable API system rather than static text matching.
Why it matters: API-Bank was one of the first benchmarks to pair a runnable evaluation harness (actual callable APIs, not just string-matched outputs) with a companion large-scale training set, letting researchers both measure and directly improve tool-use ability.
Known limitations: Published in 2023, before today's native function-calling APIs were standard, so newer models may benefit from tool-calling formats or training the original evaluation harness wasn't designed around.
What API-Bank Measures
API-Bank's evaluation set consists of 314 tool-use dialogues annotated with 753 API calls across a runnable system of 73 API tools, explicitly designed to test three progressively harder questions: how effective current LLMs are at using tools, how to improve tool-use ability, and what obstacles remain. Because the APIs are actually runnable (not simulated via text matching), models must produce well-formed, executable calls rather than merely plausible-looking ones.
To support the second question — improving tool-use ability — the authors also released a much larger training set of 1,888 tool-use dialogues spanning 2,138 APIs across 1,000 distinct domains, which they used to train Lynx, a tool-augmented model initialized from Alpaca. The paper reports that GPT-3.5 showed improved tool utilization over GPT-3, GPT-4 excelled specifically at planning, and Lynx surpassed Alpaca's baseline tool-use performance by more than 26 points, approaching GPT-3.5-level effectiveness.
Benchmark Specifications
| Field | Value |
|---|---|
| Task category | Agentic / tool-augmented LLM evaluation |
| Metric | Accuracy across planning, retrieval, and API-calling stages |
| Number of tasks | 314 annotated dialogues, 753 API calls, 73 runnable API tools |
| Saturation | Medium — GPT-4 leads on planning but "significant potential for improvement" remains per the paper |
| Created by | Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, Yongbin Li (DAMO Academy, Alibaba Group) |
| Source paper | Li et al. 2023 (EMNLP 2023) |
| GitHub | AlibabaResearch/DAMO-ConvAI (api-bank) |
How API-Bank Is Scored
Accuracy is measured at each stage of the tool-use pipeline: whether the model correctly decides an API call is needed (planning), whether it selects the right API from the candidate pool (retrieval), and whether the final call it issues executes successfully with correct parameters (calling). Aggregate accuracy across the 753 annotated calls is the headline metric, with per-stage breakdowns used to diagnose where a model's tool-use pipeline actually fails.
State-of-the-Art Results
Scores vary by model generation and tool-calling interface used. See the official GitHub repository for the paper's full per-model breakdown across planning, retrieval, and calling stages.
API-Bank on Benchgen
No Benchgen results yet — be the first to run API-Bank.
API-Bank vs Other Benchmarks
| Benchmark | What it tests | Tasks | Saturation |
|---|---|---|---|
| API-Bank | Runnable tool planning, retrieval, and calling | 314 dialogues | medium |
| Tau-bench | Simulated customer-service tool-use with user agent | — | medium |
| BFCL | Function-calling format accuracy | — | high |
API-Bank is distinguished by its runnable API execution model (real callable tools, not just format-matching) combined with a companion large-scale training set, making it as much a training resource as an evaluation benchmark.
Run API-Bank on Your Model
Benchgen lets teams evaluate their own model's tool-use pipeline against API-Bank's runnable API suite, breaking down accuracy by planning, retrieval, and calling stage to pinpoint exactly where tool-use failures occur.