Research map / Multi-agent systems and games / LLM agents
T1: A Tool-Oriented Conversational Dataset for Multi-Turn Agentic Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces T1, a 13.5k-dialogue multi-turn, multi-domain travel-planning tool-use dataset with a caching mechanism, and T1-Agent, a code-generating agent evaluated on open-weight and proprietary LLMs, with SFT of Llama 3.1 8B.
Problem
Planning tool calls with inter-tool dependencies across multi-turn, multi-domain conversations lacks adequate training and evaluation resources; existing tool-use datasets are mostly single-turn or single-domain.
Approach
A synthetic, template-based multi-turn travel dialogue dataset with executable ground-truth code over 14 tools, plus a cache-summary mechanism in the agent that lets the LLM reuse prior tool results during planning; SFT of an 8B model is shown to match a 70B model.
Key results
- Llama 3.1 8B Instruct SFT reaches 77.58% tool-call accuracy and 87.17 tool-call F1 on the large split, outperforming Llama 3.3 70B Instruct (67.39 accuracy, 79.72 F1).
- On the small split, GPT 5 achieves 94.45% code execution rate and 76.25 cache-summary EM, while GPT 4.1-nano lags with 63.14 tool-call accuracy.
- Llama 3.3 70B Instruct attains 91.43% code execution accuracy on the large split, the highest among open-weight models there, versus 84.29% for the SFT 8B model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Llama 3.1 8B Instruct on the T1 training set (ablation) | Llama 3.1 8B Instruct · T1 large | Tool Call F1: 52.11 -> 87.17 | Helped |
| Fine-tune Llama 3.1 8B Instruct on the T1 training set (ablation) | Llama 3.1 8B Instruct · T1 large | Parameter Matching F1: 31.79 -> 75.76 | Helped |
| Fine-tune Llama 3.1 8B Instruct on the T1 training set (ablation) | Llama 3.1 8B Instruct · T1 small | Tool Call F1: 52.25 -> 87.07 | Helped |
| Increase few-shot examples from 0 to 5 and 13 (ablation) | Llama 3.3 70B Instruct · T1 large flights | Tool Call F1 | Helped |
Surprising findings
- The SFT-tuned 8B model, without explicit reasoning prompts, matches or exceeds the 70B instruct model on tool-call metrics.
Limitations
- Evaluation limited to open-weight models in the main large-split experiments; proprietary models only on the small split due to resource constraints.
- No error bars or statistical significance reported due to computational cost.
- Dataset is synthetic and restricted to US travel domain with template-based dialogues.
Experimental setup
Datasets: T1
Benchmarks: T1
Metrics: accuracy precision recall F1 code execution rate SacreBLEU BERTScore F1 exact match
Baselines: Llama 3.1 8B Instruct Llama 3.3 70B Instruct Phi-4-reasoning-plus s1.1 32B Gemini 2.5 Pro OpenAI o3 OpenAI o4-mini GPT 4.1 GPT 4.1-mini GPT 4.1-nano GPT 5
Builds on: LoRA ReAct TravelPlanner
Closest papers by meaning
- ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interaction (ICLR 2026 · similarity 0.91)
- ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning (NeurIPS 2026 · similarity 0.89)
- Large Language Models as Planning Domain Generators (Student Abstract) (AAAI 2024 · similarity 0.89)
- Joint Optimization of Tool Creation and Use for Large Language Model Agents (NeurIPS 2026 · similarity 0.89)
- CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI (NeurIPS 2026 · similarity 0.88)
- ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.