Research map / Large language models / Agents, code and math

Reducing Tool Hallucination via Reliability Alignment

ICML 2025 · Large language models › Agents, code and math · approach: Agent instruction-following benchmarks · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Defines two types of tool hallucination (selection and usage), builds RelyToolBench with missing-parameter and unmatched-tool subsets, and proposes Relign, which adds indecisive actions (ChangeTools, TalkToUser) and trains with SFT then DPO to reduce tool hallucinations.

Problem

LLM agents often hallucinate tool use: they select inappropriate tools or misuse them (fabricated or missing parameters), causing erroneous task execution, extra cost, and unreliable outputs. Existing tool-learning work mainly optimizes decisive tool calls and lacks a way to defer, clarify, or switc

Approach

Expand the tool-use action space with indecisive actions (ChangeTools, TalkToUser, give up) so the model can defer or clarify when tool preconditions are unmet, and align it with SFT on synthesized boundary cases followed by DPO on preference pairs ranking correct > indecisive > hallucinated actions.

Key results

• Relign on Llama3.1-8B raises overall RePR to 77.2 from 65.3 baseline and cuts tool hallucination rate from 50.8% to 14.6%.
• Relign on ToolLLaMA lowers overall tool hallucination from 56.5% to 18.3% and raises RePR from 62.5 to 69.1.
• On APIBench OOD with BM25 retrieval, Relign-trained Llama3.1 reduces HuggingFace hallucination from 9.7 to 6.4 and raises AST from 14.3 to 15.7.

Measured effects reported in the paper

Surprising findings

• StepTool improves pass rate but does not reduce tool hallucination rate or tool call count.
• DPO alone increases hallucination relative to SFT in several settings, while SFT alone gives the largest hallucination reduction.

Limitations

• Hallucination evaluation relies on GPT-4o as judge; human check reports 92.7% accuracy on 150 cases.
• RelyToolBench is derived from StableToolBench and its unmatched-tool and missing-parameter subsets are synthetically constructed.
• Timing hallucinations show smaller improvement than other types.

Experimental setup

Datasets: ToolBench StableToolBench

Benchmarks: RelyToolBench APIBench

Metrics: RePR (reliable pass rate) tool hallucination rate average tool call number benefit-cost utility AST accuracy

Baselines: gpt-3.5-turbo GPT-4o-mini GPT-4o ToolLLaMA + RLHF StepTool SFT DPO SFT&DPO

Builds on: ToolBench StableToolBench StepTool Direct Preference Optimization

Closest papers by meaning

• CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification (AAAI 2025 · similarity 0.87)
• Exposing and Evaluating Hallucinations for GUI Grounding (CVPR 2026 · similarity 0.87)
• MCP-Atlas: A Large-Scale Benchmark for Tool-Use Competency with Real MCP Servers (NeurIPS 2026 · similarity 0.86)
• MLLMs Fail to Refuse when Using Tools Agentically (NeurIPS 2026 · similarity 0.86)
• Learning Evolving Tools for Large Language Models (ICLR 2025 · similarity 0.86)
• Beyond Accuracy: A Cognitive Load Framework for Mapping the Capability Boundaries of Tool-use Agents (AAAI 2026 · similarity 0.85)