Research map / Large language models / Agents, code and math
Advancing Tool-Augmented Large Language Models: Integrating Insights from Errors in Inference Trees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds step-wise preference pairs from ToolBench DFSDT decision trees (using failed branches) and applies SFT followed by DPO to LLaMA-2-7B, yielding TP-LLaMA for tool-augmented reasoning.
Problem
Tool-augmented LLMs trained by SFT on ToolBench use only successful DFSDT paths, discarding information from failed branches, which limits data utilization and exploration.
Approach
Parse each branch node along the successful path of a DFSDT tree into a step-wise preference pair (success child preferred over failed sibling), then run SFT on successful trajectories followed by DPO on these pairs.
Key results
- TP-LLaMA achieves an average pass rate of 0.65 versus 0.53 for LLaMA with SFT and 0.49 for ToolLLaMA on ToolBench test scenarios.
- TP-LLaMA averages 22.62 DFSDT steps versus 32.06 for LLaMA with SFT (29.44% reduction).
- Replacing the base model with Mistral-7B, Qwen1.5-7B, or Gemma-7B and applying the same pipeline improves average pass rate by at least 8% over SFT-only versions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DPO training on step-wise ToolPreference data after SFT | LLaMA-2-7B tool-use model · ToolBench six test scenarios (Avg) | pass rate: 0.53 -> 0.65 | Helped |
| Add DPO training on step-wise ToolPreference data after SFT | LLaMA-2-7B tool-use model · ToolBench six test scenarios (Avg) | win rate: 0.61 -> 0.65 | Helped |
| Add DPO training on step-wise ToolPreference data after SFT | LLaMA-2-7B tool-use model · ToolBench six test scenarios (Avg) | DFSDT inference steps: 32.06 -> 22.62 | Helped |
| Add DPO training on step-wise ToolPreference data after SFT | LLaMA-2-7B tool-use model · G3-Ins. | pass rate: 0.35 -> 0.61 | Helped |
| Add DPO training on step-wise ToolPreference data after SFT (ablation) | Mistral-7B tool-use model · ToolBench six test scenarios (Avg) | pass rate: 0.47 -> 0.62 | Helped |
| Add DPO training on step-wise ToolPreference data after SFT (ablation) | Mistral-7B tool-use model · ToolBench six test scenarios (Avg) | win rate: 0.58 -> 0.60 | Helped |
| Add DPO training on step-wise ToolPreference data after SFT (ablation) | Mistral-7B tool-use model · ToolBench six test scenarios (Avg) | DFSDT inference steps: 27.22 -> 22.39 | Helped |
| Add DPO training on step-wise ToolPreference data after SFT (ablation) | Qwen1.5-7B tool-use model · ToolBench six test scenarios (Avg) | pass rate: 0.57 -> 0.65 | Helped |
| Add DPO training on step-wise ToolPreference data after SFT (ablation) | Qwen1.5-7B tool-use model · ToolBench six test scenarios (Avg) | win rate: 0.60 -> 0.63 | Helped |
| Add DPO training on step-wise ToolPreference data after SFT (ablation) | Qwen1.5-7B tool-use model · ToolBench six test scenarios (Avg) | DFSDT inference steps: 35.68 -> 24.40 | Helped |
| Add DPO training on step-wise ToolPreference data after SFT (ablation) | Gemma-7B tool-use model · ToolBench six test scenarios (Avg) | pass rate: 0.47 -> 0.65 | Helped |
| Add DPO training on step-wise ToolPreference data after SFT (ablation) | Gemma-7B tool-use model · ToolBench six test scenarios (Avg) | win rate: 0.58 -> 0.64 | Helped |
Surprising findings
- Reported ToolLLaMA results from the authors' reproduction are lower than those in the original ToolBench paper, which the authors attribute to API distribution shift.
- Under the revised GPT-assisted pass-rate definition, the gap between TP-LLaMA and ToolLLaMA narrows considerably.
Limitations
- Performance depends on quality of expert-explored decision trees; manually inserted suboptimal branches are not explored.
- Preferences are not compared between steps on failure paths, leaving data utilization room to improve.
- Each step requires the full history as input, which is time-consuming.
Experimental setup
Datasets: ToolBench ToolPreference
Benchmarks: ToolBench test sets (G1-Ins., G1-Tool, G1-Cat., G2-Ins., G2-Cat., G3-Ins.)
Metrics: pass rate win rate average DFSDT steps
Baselines: ChatGPT+DFSDT Text-Davinci-003 ToolLLaMA LLaMA with SFT
Builds on: ToolBench DFSDT Direct Preference Optimization ToolLLaMA
Closest papers by meaning
- TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees (ICLR 2025 · similarity 0.90)
- AlphaZero-Like Tree-Search can Guide Large Language Model Decoding and Training (ICML 2024 · similarity 0.90)
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.90)
- Enhancing Decision-Making of Large Language Models via Actor-Critic (ICML 2025 · similarity 0.89)
- Advancing LLM Reasoning Generalists with Preference Trees (ICLR 2025 · similarity 0.89)
- Automated Creation of Reusable and Diverse Toolsets for Enhancing LLM Reasoning (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.