Research map / Multi-agent systems and games / LLM agents
AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AnyTool, a GPT-4-driven agent that retrieves from a 16K+ RapidAPI pool via a hierarchical agent retriever and solves queries with a self-reflective loop, plus a revised evaluation protocol and the AnyToolBench benchmark.
Problem
Tool use with very large API pools (16K+ RapidAPI APIs) exceeds LLM context limits and retrieval is inaccurate, while the existing ToolBench evaluation protocol inflates pass rates for unsolvable queries.
Approach
A training-free, three-tier hierarchical agent retriever (meta-agent, category agents, tool agents) built on GPT-4 function calling, combined with a DFSDT solver and a self-reflection loop that re-activates retriever agents and the solver using failure reasons; the evaluation is revised to drop the solvable/non-solvable phase.
Key results
- AnyTool reaches 58.2% average pass rate on filtered ToolBench versus 31.8% for GPT-4 w/ DFSDT using reference APIs and 22.9% for ToolLLaMA w/ DFSDT, a +32.6 point gain over ToolLLM.
- On AnyToolBench, AnyTool achieves 73.8% pass rate versus 36.6% for ToolLLM with GPT-4 solver and 14.0% for GPT-4 plain agent retriever.
- GPT-4 judge agrees with human evaluators 96.5% of the time versus 73.9% for GPT-3.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use AnyTool's self-reflective hierarchical retriever and solver instead of ToolLLM | ToolLLM on filtered ToolBench · Filtered ToolBench | average pass rate: 22.9 -> 58.2 | Helped |
| use AnyTool's self-reflective hierarchical retriever and solver instead of ToolLLM with a GPT-4 solver | ToolLLM on AnyToolBench · AnyToolBench | pass rate: 36.6 -> 73.8 | Helped |
| add a hierarchical API retriever (ablation) | AnyTool · Filtered ToolBench G2-I and G3-I | pass rate: 22.4 vs 58.9; 15.8 vs 63.2 | Helped |
| add self-reflection (ablation) | AnyTool · Filtered ToolBench G2-I and G3-I | pass rate: 19.6 vs 58.9; 15.8 vs 63.2 | Helped |
| replace DFSDT with CoT (ablation) | AnyTool · Filtered ToolBench G2-I and G3-I | pass rate: 58.9 -> 50.5; 63.2 -> 60.3 | Hurt |
| reactivate all agent levels during self-reflection instead of only tool agents (ablation) | AnyTool · Filtered ToolBench G2-I and G3-I | pass rate: 43.9 -> 58.9; 44.7 -> 63.2 | Helped |
| reactivate the meta-agent in addition to tool and category agents (ablation) | AnyTool · Filtered ToolBench G2-I and G3-I | pass rate: 55.2 -> 58.9; 55.3 -> 63.2 | Helped |
| increase the API pool from 1,000 through 5,000 and 10,000 to all APIs (ablation) | AnyTool · Filtered ToolBench G2-I and G3-I | pass rate: 18.6; 26.3; 38.1; 58.9 / 7.9; 23.7; 36.8; 63.2 | Helped |
| increase the maximum API-candidate pool size from 16 to 32 or 64 (ablation) | AnyTool · Filtered ToolBench G2-I and G3-I | pass rate: 49.5; 58.9; 58.9 / 42.1; 55.3; 63.2 | Mixed |
| set the maximum tools per tool agent to 5 instead of 3 or 10 (ablation) | AnyTool · Filtered ToolBench G2-I and G3-I | pass rate: 48.6; 58.9; 52.3 / 42.1; 57.9; 39.5 | Mixed |
Surprising findings
- Under the original ToolBench protocol, randomly selected API candidates yield an inflated 99.0% pass rate because GPT-4 labels queries non-solvable.
- A plain GPT-4 agent retrieving from the full API pool performs worse (15.2%) than GPT-4 given reference APIs, showing naive retrieval over 16K APIs fails.
Limitations
- Relies on the stability and cost of GPT-4 function calling; runtime is unstable and average consumption is about 135K tokens per query.
- Evaluation is limited by the absence of complex datasets; performance in extremely complex scenarios is unverified.
- Filtered benchmarks are small (e.g., 38 to 142 queries per subset) and manually curated.
Experimental setup
Datasets: ToolBench
Benchmarks: ToolBench (filtered) AnyToolBench
Metrics: pass rate
Baselines: ToolLLM (ToolLLaMA w/ DFSDT) GPT-4 w/ DFSDT with reference APIs GPT-4 plain agent retriever AutoGen-RAG GPT-4 GPT-3.5 w/ DFSDT
Builds on: ToolLLM DFSDT Chain of Thought AutoGen
Closest papers by meaning
- ShortcutsBench: A Large-Scale Real-world Benchmark for API-based Agents (ICLR 2025 · similarity 0.88)
- The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution (ICLR 2026 · similarity 0.84)
- APIGen: Automated PIpeline for Generating Verifiable and Diverse Function-Calling Datasets (NeurIPS 2024 · similarity 0.84)
- AutoTool: Efficient Tool Selection for Large Language Model Agents (AAAI 2026 · similarity 0.84)
- ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (ICLR 2024 · similarity 0.83)
- MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.