Research map / Large language models / Language, safety and interpretability

ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Faithfulness of LLM explanations · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds ToolDial, a 11,111-dialogue multi-turn tool-augmented dialogue dataset generated from an API graph over RapidAPI APIs with 16 user/system actions, and benchmarks LLMs on dialogue state tracking, action prediction, and faithfulness.

Problem

Existing TALM benchmarks feature simplistic, mostly single-turn dialogues that lack clarifying questions, requests for missing parameters, and fallback to additional API calls when users cannot supply required information.

Approach

Generate multi-turn tool dialogues by sampling connected API pairs from an LLM-built API graph (edges mean one API's output can feed another's input), choosing a dialogue action sequence as skeleton, augmenting it with dialogue states, and then realizing utterances with GPT-4o.

Key results

• TD-Llama (fine-tuned Llama3-8B) reaches 77.5% action prediction accuracy with ground truth and 91.0% without, outperforming GPT-4o (57.6% and 63.7%).
• GPT-4o achieves 81.4% DST accuracy with ground truth but drops to 67.8% without ground truth; TD-Llama gets 92.7% and 72.2%.
• Overall full-dialogue reasoning trace accuracy for TD-Llama is about 28.3% across 1,166 test dialogues.

Measured effects reported in the paper

Surprising findings

• GPT models rarely predict Request and Clarify actions, often rushing to answer instead of gathering information.
• TD-Llama's faithfulness (88.4%) is lower than GPT models (over 95%), indicating small models hallucinate more in responses.

Limitations

• About 30% of API graph edges are invalid (edge precision 70.9%), and dialogues are generated by GPT-4o with template-based instructions.
• Dialogues are limited to at most two chained APIs, and the dataset is English only.
• Edge validation relies on StableToolBench simulated API outputs rather than real API calls.

Experimental setup

Datasets: ToolDial RapidAPI APIs

Benchmarks: ToolDial

Metrics: dialogue state tracking accuracy action prediction accuracy per-action F1 faithfulness score

Baselines: GPT-3.5-turbo GPT-4o-mini GPT-4-turbo GPT-4o CodeLlama-7b-Instruct-hf Qwen2.5-Coder-7B-Instruct Llama3-8B-Instruct

Builds on: ToolBench API-Bank ToolTalk StableToolBench Schema-Guided Dialogue

Closest papers by meaning

• ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interaction (ICLR 2026 · similarity 0.88)
• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.88)
• T1: A Tool-Oriented Conversational Dataset for Multi-Turn Agentic Planning (NeurIPS 2025 · similarity 0.88)
• ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents (ICLR 2025 · similarity 0.87)
• ToolACE: Winning the Points of LLM Function Calling (ICLR 2025 · similarity 0.87)
• MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (ICLR 2024 · similarity 0.87)