Research map / Multi-agent systems and games / LLM agents

TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarios

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces TRIP-Bench, a long-horizon travel-planning benchmark with tools, rubric-based constraints and hard interaction splits, plus GTPO, an online multi-turn RL method that lifts Qwen2.5-32B-Instruct above Gemini-3-Pro under the paper's evaluation.

Problem

Existing agent benchmarks underrepresent global constraint enforcement, multi-tool reasoning, and adaptation to evolving user behavior over long multi-turn interactions.

Approach

A travel-planning benchmark built from rubric-based constraint generators/validators, progressive modification chains, and difficulty-controlled user simulators (LIT, FIT, AIS, PMR), paired with GTPO, a turn-level group RL method using global instruction normalization, turn-wise reward differencing, and per-turn reward normalization.

Key results

• Best strict overall score is GPT-5.2 (w/ thinking) at 18.5%; best loose overall is 45%, and hard-split strict scores are mostly below 10%.
• GTPO on Qwen2.5-32B-Instruct reaches 49 loose / 21 strict on Easy, versus SFT at 32 / 3, and 40 loose on Mid.
• Thinking mode raises DeepSeek-V3.2 Overall strict from 2.3 to 10.5.

Surprising findings

• Multi-turn interaction underperforms single-turn on complex tasks with strong global constraints, with a gap of over 20 points on the hardest sets under loose scoring.
• DeepSeek-V3.2 with thinking achieves loose performance comparable to GPT-5.2 at roughly 10% of the cost.

Limitations

• Hard split often needs context beyond 128k tokens, so trained models were evaluated only on Easy and Mid.
• Strict scores remain near zero on hard subsets, including FIT, for all evaluated models.
• Only 20 trajectories were used for manual reliability check of the user simulator.

Experimental setup

Datasets: TripTailor (expanded) Toucan

Benchmarks: TRIP-Bench

Metrics: strict overall success loose overall success pass@k

Baselines: GRPO SFT Gemini-3-Pro GPT-5.2 Claude-Sonnet-4.5 DeepSeek-V3.2 Kimi-K2 Qwen3-235B-A22B GLM-4.7 Gemini-3-Flash

Builds on: GRPO PPO TripTailor tau2-Bench

Closest papers by meaning

• Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning (NeurIPS 2026 · similarity 0.88)
• WorldTravel: A Realistic Multimodal Travel-Planning Benchmark with Tightly Coupled Constraints (ICML 2026 · similarity 0.88)
• $\texttt{Multi}^2$: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environments (ICML 2026 · similarity 0.87)
• Group-in-Group Policy Optimization for LLM Agent Training (NeurIPS 2025 · similarity 0.87)
• Agentic Reinforced Policy Optimization (ICLR 2026 · similarity 0.87)
• ATLAS: Constraints-Aware Multi-Agent Collaboration for Real-World Travel Planning (ICLR 2026 · similarity 0.87)