Research map / Large language models / Agents, code and math
TravelPlanner: A Benchmark for Real-World Planning with Language Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TravelPlanner, a sandboxed multi-day travel-planning benchmark with 1,225 queries and six search tools, showing that even GPT-4-Turbo with ReAct passes only 0.6% of plans.
Problem
Existing language-agent planning benchmarks use single-objective optimization with fixed ground truths, so it is unclear whether LLM agents can plan in complex, multi-constraint, real-world settings such as travel itineraries that require tool use and commonsense reasoning.
Approach
A closed, static sandbox of about four million travel records accessed through six search tools, with 1,225 queries spanning 3/5/7-day trips and easy/medium/hard constraint levels, evaluated with automatic scripts on commonsense, hard-constraint, and final pass rates in two-stage (tool-use plus planning) and sole-planning modes.
Key results
- GPT-4-Turbo with ReAct in two-stage mode achieves only 0.6% final pass rate on the test set, and no other LLM passes any task in that mode.
- In sole-planning mode, Direct GPT-4-Turbo reaches 4.4% final pass rate on test, with 80.6% micro commonsense pass rate and 44.3% micro hard-constraint pass rate.
- Human annotators take about 12 minutes per plan while GPT-3.5-Turbo agents produce a plan in 1-2 minutes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide the necessary planning information directly instead of requiring tool-based information collection | GPT-4-Turbo on TravelPlanner · TravelPlanner test | Final Pass Rate: 0.6 -> 4.4 | Helped |
| Use chain-of-thought instead of direct planning | GPT-3.5-Turbo · TravelPlanner test, sole-planning | Final Pass Rate: 0.6 -> 0.4 | Hurt |
| Use ReAct cost feedback instead of direct planning | GPT-3.5-Turbo · TravelPlanner test, sole-planning | Final Pass Rate: 0.6 -> 0.7 | Helped |
| Use Reflexion instead of direct planning | GPT-3.5-Turbo · TravelPlanner test, sole-planning | Final Pass Rate: 0.6 -> 0.6 | No clear effect |
Surprising findings
- Two-stage performance is lower than sole-planning performance across all metrics, with gaps over 30 points, suggesting multitasking with information collection degrades planning.
- Stronger planning strategies such as ReAct and Reflexion did not outperform the Direct baseline in sole-planning mode for GPT-3.5-Turbo.
Limitations
- The sandbox is static and built partly from 2022 data and synthetic modifications, so it may not reflect live travel conditions.
- Commonsense criteria reflect the authors' consensus and may not match all definitions of commonsense.
- Evaluation covers a limited set of LLMs and planning strategies, and ToT and GoT were excluded due to cost.
Experimental setup
Datasets: Kaggle Flight Status Prediction dataset (2022) Google Distance Matrix API data Kaggle Zomato Restaurants dataset Google Places API data Kaggle Airbnb Open Data
Benchmarks: TravelPlanner validation set (180 queries) TravelPlanner test set (1000 queries)
Metrics: delivery rate commonsense constraint pass rate (micro/macro) hard constraint pass rate (micro/macro) final pass rate
Baselines: Greedy Search Mistral-7B-32K Mixtral-8x7B-MoE Gemini Pro GPT-3.5-Turbo GPT-4-Turbo ZS-CoT ReAct Reflexion
Builds on: ReAct Reflexion Chain-of-thought prompting
Closest papers by meaning
- LLMs Can Plan Only If We Tell Them (ICLR 2025 · similarity 0.90)
- LexiCon: a Benchmark for Planning under Temporal Constraints in Natural Language (NeurIPS 2025 · similarity 0.90)
- Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks (ICML 2025 · similarity 0.90)
- ATLAS: Constraints-Aware Multi-Agent Collaboration for Real-World Travel Planning (ICLR 2026 · similarity 0.89)
- Large Language Models as Planning Domain Generators (Student Abstract) (AAAI 2024 · similarity 0.89)
- SPIRAL: Symbolic LLM Planning via Grounded and Reflective Search (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.