Research map / Large language models / Agents, code and math
ChinaTravel: An Open-Ended Travel Planning Benchmark with Compositional Constraint Validation for Language Agents
Jie-Jing Shao, Bo-Wen Zhang, Xiao-Wen Yang, Baizhi Chen, Siyu Han, Pang Jinghao, Wen-Da Wei, Guohao Cai, Zhenhua Dong, Lan-Zhe Guo, Yu-Feng Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ground-truth (oracle) DSL instead of LLM NL2DSL translation (ablation) | NeSy Planning (DeepSeek-V3) · ChinaTravel Easy | Final Pass Rate: 52.6 -> 74.0 | Improved |
| use ground-truth DSL instead of LLM translation (ablation) | NeSy Planning (DeepSeek-V3) · ChinaTravel Human-Val | Final Pass Rate: 37.0 -> 45.4 | Improved |
| use ground-truth DSL instead of LLM translation (ablation) | NeSy Planning (DeepSeek-V3) · ChinaTravel Human-Test | Final Pass Rate: 23.3 -> 31.6 | Improved |
| replace pure-LLM ReAct with neuro-symbolic planning | DeepSeek-V3 · ChinaTravel Easy | Final Pass Rate: 5.33 -> 52.6 | Improved |
| replace ReAct with NeSy Planning | GPT-4o · ChinaTravel Easy | Final Pass Rate: 0 -> 60.6 | Improved |
| use NeSy Planning instead of TTG MILP solver | DeepSeek-V3 · ChinaTravel Easy | Final Pass Rate: 8.66 -> 52.6 | Improved |
| add LLM-modulo oracle verifier feedback loop (10 rounds) | LLM-based planning · ChinaTravel Easy | Final Pass Rate: 4.33 (DeepSeek), 7.00 (GPT-4o) | No clear effect |
| add 1-shot example to ReAct | DeepSeek-V3 · ChinaTravel Human-Val | Final Pass Rate: 0 -> 2.59 | No clear effect |
| add syntax-checking Reflexion to NL2DSL | LLMs in NeSy Planning · DSL translation | constraint count | Hurt |
| use LLM-enhanced preference recommendation (PEQ) over baseline (BQ) (ablation) | NeSy Planning · single-preference subsets | preference optimization: better in 5/6 scenarios | Mixed |
| use PEQ over BQ for two preferences (ablation) | NeSy Planning · multi-preference trade-offs | Pairwise Pareto win rate: 0.58 (P3&P4), 0.45 (P1&P2), 0.10 (P1&P4) | Mixed |
| move from literal to semantic POIs in intent grounding | DeepSeek-V3 / GPT-4o · Intent Grounding | accuracy: DS 94% -> 76%, GPT 79% -> 53% | Hurt |
Closest papers by meaning
- TravelPlanner: A Benchmark for Real-World Planning with Language Agents (ICML 2024 · similarity 0.91)
- WorldTravel: A Realistic Multimodal Travel-Planning Benchmark with Tightly Coupled Constraints (ICML 2026 · similarity 0.89)
- ATLAS: Constraints-Aware Multi-Agent Collaboration for Real-World Travel Planning (ICLR 2026 · similarity 0.88)
- TraveLLaMA: A Multimodal Travel Assistant with Large-Scale Dataset and Structured Reasoning (AAAI 2026 · similarity 0.87)
- TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarios (ICML 2026 · similarity 0.86)
- Wide-Horizon Thinking and Simulation-Based Evaluation for Real-World LLM Planning with Multifaceted Constraints (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
