Tau2 Airline — Results
| Rank | Model | Score |
|---|---|---|
| 1 | longcat-flash-thinking-2601 | 0.765 |
| 2 | nova-2-omni | 0.688 |
| 3 | longcat-flash-thinking | 0.675 |
| 4 | gpt-5-1-instant | 0.67 |
| 5 | gpt-5-1-thinking | 0.67 |
| 6 | gpt-5-1 | 0.67 |
| 7 | nova-2-pro | 0.652 |
| 8 | nova-2-lite | 0.648 |
| 9 | o3 | 0.648 |
| 10 | claude-haiku-4-5 | 0.636 |
Tau2 Airline
1 phaseActive
τ²-Bench airline domain — evaluates conversational agents in a dual-control setup on rebooking, refund, and policy-constrained airline support tasks. Metric: success rate.
Overview
Tau2 Airline
Quick answer: Tau2 Airline is the airline domain of τ²-Bench, which evaluates conversational agents in a dual-control environment — where both the AI agent and the simulated user can invoke tools — on rebooking, refund, and policy-constrained airline customer-support scenarios. LongCat-Flash-Thinking-2601 leads at a 76.5% success rate, well below saturation.
What Does Tau2 Airline Test?
Airline customer support is a demanding testbed for dual-control agents: policies are strict (change fees, fare rules, cancellation windows), actions are often irreversible (rebooking a flight, issuing a refund), and the user frequently has to perform steps themselves (confirming a new itinerary, reading back a confirmation code). τ²-Bench models this as a Dec-POMDP where both the agent and a simulated user act on a shared environment via tools, testing whether the agent can coordinate the interaction rather than just calling the right API.
| Focus area | Examples |
|---|---|
| Policy compliance | Enforcing fare rules, change fees, and refund eligibility exactly as written |
| Dual-control coordination | Agent and user both act on the shared airline reservation system |
| Irreversible-action caution | Confirming details before rebooking, cancelling, or refunding |
| Communication | Giving the user clear, correct instructions for their side of the task |
How Is Tau2 Airline Scored?
Each task is graded by comparing the final environment state — across both agent-controlled and user-controlled actions — against a ground-truth expected outcome. A task is scored as fully successful (1) or failed (0); the reported score is the average success rate across the task suite, normalized to 0–1.
Key Facts
| Property | Value |
|---|---|
| Released | June 2025 |
| Created by | Sierra Research |
| Domain | Airline customer support (dual-control) |
| Metric | Success rate |
| Score range | 0–1 |
| Top model | LongCat-Flash-Thinking-2601 (0.765) |
Tau2 Airline on Benchgen
No Benchgen results yet — be the first to run Tau2 Airline.
Tau2 Airline vs Other Benchmarks
| Benchmark | What it tests | Saturation |
|---|---|---|
| Tau2 Airline | Dual-control airline customer support | Low |
| Tau2 Retail | Dual-control retail customer support | Medium-high |
| Tau2 Telecom | Dual-control telecom troubleshooting | High |
FAQ
What is Tau2 Airline? Tau2 Airline is the airline domain of τ²-Bench, a benchmark evaluating conversational AI agents in a dual-control environment where both the agent and a simulated user use tools to jointly resolve airline customer-support tasks like rebooking, refunds, and policy-constrained changes.
Who created Tau2 Airline? τ²-Bench was created by Sierra Research — Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and collaborators — published in June 2025, as a successor to the original τ-bench (Tau-bench Airline).
How is τ²-Bench different from τ¹ (Tau-bench Airline)? The original τ-bench only lets the agent use tools while the user is a passive information provider. τ²-Bench models the interaction as a dual-control Dec-POMDP, where the user must also take tool-based actions — a more realistic simulation of airline support where the customer confirms bookings or reads back details on their own end.
What score does the best model achieve on Tau2 Airline? LongCat-Flash-Thinking-2601 leads at 0.765 (76.5%), followed by Amazon Nova 2 Omni at 0.688.
Is Tau2 Airline saturated? No. The top score is only 0.765, well below the near-saturated results seen on Tau2 Telecom — airline's strict policy constraints and irreversible actions make it one of the harder τ²-Bench domains.