TAU-bench Airline — Results
| Rank | Model | Score |
|---|---|---|
| 1 | claude-sonnet-4-5 | 0.7 |
| 2 | minimax-m1-80k | 0.62 |
| 3 | glm-4-5-air | 0.608 |
| 4 | glm-4-5 | 0.604 |
| 5 | claude-sonnet-4 | 0.6 |
| 6 | minimax-m1-40k | 0.6 |
| 7 | qwen3-coder-480b-a35b | 0.6 |
| 8 | claude-opus-4 | 0.596 |
| 9 | claude-opus-4-1 | 0.56 |
| 10 | gpt-4-5 | 0.5 |
| 11 | o1 | 0.5 |
| 12 | gpt-4-1 | 0.494 |
| 13 | o4-mini | 0.492 |
| 14 | qwen3-next-80b-a3b-thinking | 0.49 |
| 15 | claude-3-5-sonnet | 0.46 |
| 16 | qwen3-235b-a22b-thinking-2507 | 0.46 |
| 17 | qwen3-next-80b-a3b-instruct | 0.44 |
| 18 | gpt-4o | 0.428 |
| 19 | gpt-4-1-mini | 0.36 |
| 20 | o3-mini | 0.324 |
| 21 | claude-3-5-haiku | 0.228 |
TAU-bench Airline
1 phaseActive
Evaluates language agents on airline domain conversations — booking changes, rule-following, and dynamic dialogue via API tools. Metric: success rate.
Overview
TAU-bench Airline
Quick answer: TAU-bench Airline is the airline domain of τ-bench (TAU-bench), evaluating language agents' ability to interact with users through dynamic conversations while following airline-specific rules and using API tools for tasks like booking changes and cancellations. Claude Sonnet 4.5 leads with 70.0% across 23 evaluated models — notably harder than the retail domain.
What Does TAU-bench Airline Test?
TAU-bench Airline evaluates tool-agent-user interaction in an airline customer-service setting, requiring agents to interpret user requests, apply strict airline policy rules (e.g., fare rules, baggage allowances, rebooking windows), and invoke the correct sequence of API tools — all while maintaining a coherent multi-turn conversation with a simulated user.
| Focus area | Examples |
|---|---|
| Multi-turn dialogue | Sustaining context across a realistic airline support conversation |
| Tool calling | Invoking airline APIs (flight search, rebooking, cancellation) |
| Policy adherence | Following complex fare, baggage, and rebooking rules |
| User simulation | Agent is evaluated against an LLM-simulated user with hidden intents |
How Is TAU-bench Airline Scored?
Each task is graded by comparing the final environment/database state after the conversation to a ground-truth expected state, plus verifying required user communications occurred. A task is scored as fully successful (1) or failed (0); the reported score is the average success rate (pass^1) across the task suite, normalized to 0–1.
Key Facts
| Property | Value |
|---|---|
| Released | June 2024 |
| Created by | Sierra Research |
| Domain | Airline customer service |
| Metric | Success rate |
| Score range | 0–1 |
| Top model | Claude Sonnet 4.5 (0.700) |
| Models evaluated | 23 |
FAQ
What is TAU-bench Airline? TAU-bench Airline is the airline domain of τ-bench, a benchmark for tool-agent-user interaction that evaluates language agents on realistic airline support conversations requiring strict rule-following and tool use.
Who created TAU-bench Airline? TAU-bench (τ-bench) was created by Sierra Research — Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan — published in June 2024.
Why is TAU-bench Airline harder than TAU-bench Retail? Airline policies involve more complex, interacting rules (fare classes, baggage, rebooking windows, multi-passenger itineraries), and even top models plateau well below saturation — the current average score across all models is just 0.5, versus 0.7 for retail.
What score does the best model achieve on TAU-bench Airline? Claude Sonnet 4.5 leads with 0.700 (70.0%), followed by MiniMax M1 80K at 0.620 and Zhipu AI's GLM-4.5-Air at 0.608.
Is TAU-bench Airline saturated? No. With a leader at just 0.700 and an average of 0.5 across 23 models, this remains one of the harder agentic benchmarks, with substantial room for improvement.