Want to try it first? A free 10-task-per-domain sample is available on the Demo Samples page.
What Tau2-Bench is
Each sample is a conversation between a customer and an AI service agent. The agent is given a domain policy and a fixed tool set, and it must satisfy the customer’s request — booking a flight, troubleshooting a phone, returning an order — by calling backend tools while staying inside policy. A user simulator plays the customer across many turns, supplying details only when asked, changing its mind, and pushing back, so the agent has to drive a real conversation rather than answer a single prompt. Three properties make the tasks hard:- Policy compliance. The agent must obtain explicit confirmation before any write, refuse out-of-policy requests, and transfer to a human when (and only when) a request falls outside its tools.
- Multi-turn user simulation. Customers reveal intent gradually, batch several requests together, and frequently switch to a fallback goal when the first one is not allowed.
- Stateful environments. Tools read and write a shared database, so a correct final answer requires a correct sequence of actions, not just a correct last message.
Task categories
At a glance
What’s inside
Trajectory length
Tau2-Bench dialogs are genuinely long-horizon. The table below summarizes the SFT dialogs per domain — messages per dialog (system + user + assistant + tool), user-simulator turns, and agent tool calls — as mean / median / p90.- Telecom dialogs are the longest (median 72 messages, ~30 user turns) but use the fewest tool calls — troubleshooting is a slow, conversational back-and-forth where each device check is interpreted before the next.
- Airline dialogs make the most tool calls (median 11) — booking and modification require chained lookups (user → reservations → flights) before a single write.
- Retail sits in between, with multi-request dialogs that mix lookups, calculations, and writes.
Tool usage
Each domain exercises a distinct slice of its tool set, reflecting its source material:- Airline — dominated by
get_reservation_details,search_direct_flight, andget_flight_status, withupdate_reservation_baggages,book_reservation, andupdate_reservation_flightsperforming the writes after confirmation. - Telecom — dominated by
get_details_by_idandget_customer_by_phonefor identification, then device-state actions likerefuel_data,enable_roaming, andget_data_usage;transfer_to_human_agentsappears when a fix is out of scope. - Retail — dominated by
get_order_detailsandget_product_details, withfind_user_id_by_email/get_user_detailsfor authentication andexchange_delivered_order_items,return_delivered_order_items,modify_user_address, andcancel_pending_orderperforming the changes.
How challenging is the data
The difficulty is not in any single tool call — it is in doing the whole conversation correctly. A passing trajectory must authenticate the user, gather requirements across many turns, take the exact gold sequence of state-changing actions, obtain confirmation before each write, and refuse or transfer when policy requires it. A single skipped confirmation, an out-of-policy action, or a wrong fallback when the user changes their mind fails the task. Frontier models reliably handle the read-only lookups but stumble on the end-to-end requirements — correct database mutations and full policy adherence and goal completion in one trajectory. The harder, single-domain banking slice is quantified on the Tau3-Bench page, where frontier models pass under 30% of tasks.Training utility
Training studies on the Tau2-Bench corpus show that SFT bootstrap + verifiable-reward RL lifts open-weight models to frontier-competitive performance across all three domains, using EigenData’s executable per-task verifiers as the RL reward. Results are reported as Pass^k — the probability that all k independent trials of a task succeed (a strict reliability metric, not one lucky rollout). In every case the RL training tasks are generated from environments whose databases are held out from the benchmark test set (same tool schemas and operational rules, different users / orders / products). The pattern is consistent across domains: SFT delivers the first large jump (broad tool-use competence — tool selection, schema-valid arguments, multi-turn state tracking), and a second RL stage adds further gains that widen at stricter Pass^k — the agent becomes not just more often right, but more reliably right across repeated trials.The Retail comparison below ranks against the current public τ²-Bench leaderboard. The Airline and Telecom figures were graded under the τ²-bench evaluation harness at the time of measurement; the public board has since tightened its grader (the v1.0.0 “Task Quality” release) and added newer model versions, so treat those two as same-evaluation comparisons rather than current-board rankings.
Airline
Self-evolving synthetic SFT data plus GRPO RL with executable verifiers (From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents), base Qwen3-30B-A3B-Thinking-2507:
Scaling the same recipe to Qwen3-235B-A22B-Thinking-2507 reaches 73.0 Pass^1 — matching Gemini 3.0 Pro (73.0) and exceeding GPT-5 (62.5).
Retail
A targeted error-driven RL recipe (base Qwen3-30B-A3B-Thinking-2507) trains on tasks synthesized from the model’s own recurring post-SFT failures — wrong product variant, mixed item identifiers across orders, missing confirmation or execution steps:
RL adds +6.6 Pass^1 over SFT (+28.3 over base), widening at stricter consistency (Pass^4 52.6 → 61.4). Reward design matters: an ablation to task-success-only collapses to 66.7 Pass^1 (the model exploits degenerate tool-call loops), and removing the repeated-tool-call penalty drops Pass^4 to 50.0.
How it ranks. That result places our 30B model second on the current public τ²-Bench Retail leaderboard — behind only a 397B open model, and ahead of GPT-5.2, Claude Opus 4.5, and Gemini 3 Pro:
A 30B model outranking GPT-5.2, Claude Opus 4.5, and Gemini 3 Pro — second only to a model 13× its size.
Telecom
Same self-evolving SFT + RL recipe (From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents), base Qwen3-30B-A3B-Thinking-2507:
Telecom sees the largest swing — SFT alone lifts Pass^1 from 28.5 to 85.4, and RL pushes it to 95.6 (Pass^4 70.8 → 86.0). Scaling to Qwen3-235B-A22B-Thinking-2507 reaches 98.3 Pass^1, matching or exceeding every frontier model on the leaderboard (Claude Sonnet 4.5 and Gemini 3.0 Pro at 98.0, GPT-5 at 95.8).