Research map / Multi-agent systems and games / LLM agents

AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AgenTracer, an automated pipeline (counterfactual replay plus programmatic fault injection) that builds the AgenTracer-2.5K failure-attribution dataset, and trains a Qwen3-8B failure tracer with multi-granular GRPO rewards that beats larger proprietary LLMs on Who&When and gives gains to off-the-shelf multi-agent systems.

Problem

Identifying which agent and which step caused failure in long multi-agent execution traces is very hard for current LLMs (accuracy often below 10%), and existing annotated failure datasets are small.

Approach

Automatically annotate failed and synthetically corrupted trajectories with decisive-error labels via analyzer-driven counterfactual replay and fault injection, then train a small tracer with RL using a gated reward combining agent-level correctness and a Gaussian step-proximity reward.

Key results

• On Who&When handcrafted (w/ G), AgenTracer reaches 69.10% agent-level and 20.68% step-level accuracy, above Claude-Sonnet-4 (56.90% / 17.24%).
• On AgenTracer-2.5K agentic subset (w/ G), step-level accuracy is 36.17% versus 30.33% for Claude-Sonnet-4 and 27.13% for DeepSeek-R1.
• Feedback from AgenTracer improves off-the-shelf systems by up to about 14% (e.g., MaAS on MATH-500 +14.21%, OWL on GAIA +4.8%).

Surprising findings

• Providing ground-truth answers does not consistently improve attribution accuracy and sometimes lowers it (e.g., Qwen3-Coder 51.72% w/ G vs 60.35% w/o G on Who&When handcrafted).
• CRITIC feedback from GPT-4.1 degrades MaAS performance on GAIA across iterations, while conventional reflection gives no meaningful gains.

Limitations

• Analyzer and perturbation operator rely on DeepSeek-R1, so annotation quality depends on a large teacher model.
• Step-level accuracy stays low in absolute terms (e.g., 20.68% on handcrafted Who&When).
• Evaluated on a limited set of multi-agent systems and benchmarks; test splits are small (63 to 147 trajectories).

Experimental setup

Datasets: AgenTracer-2.5K Who&When MBPP+ KodCode Blackjack GAIA MATH GSM8K HotpotQA

Benchmarks: Who&When AgenTracer-2.5K code subset AgenTracer-2.5K math subset AgenTracer-2.5K agentic subset GAIA HumanEval+ MATH-500

Metrics: agent-level accuracy step-level accuracy task accuracy

Baselines: Qwen3-8B Llama-3.2-3B Qwen3-32B Qwen3-Coder GPT-4.1 DeepSeek-R1 Gemini-2.5-Pro Claude-Sonnet-4 Self-Refine CRITIC

Builds on: GRPO Who&When MAST

Closest papers by meaning

• Beyond LLM-Based Reasoning: Lightweight GNNs for Agent Failure Attribution (NeurIPS 2026 · similarity 0.92)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems (ICLR 2026 · similarity 0.91)
• Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems (ICML 2025 · similarity 0.90)
• Who&When Pro: Can LLMs Really Attribute Failures in AI Agents? (NeurIPS 2026 · similarity 0.90)
• Why Do Multi-Agent LLM Systems Fail? (NeurIPS 2025 · similarity 0.89)
• CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems (ICML 2026 · similarity 0.89)