Research map / Multi-agent systems and games / LLM agents

AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces online auditing for multi-agent systems, where an auditor issues continue-or-alarm verdicts on each unfolding prefix, plus the AFTraj-2K corpus and a coarse-to-fine (BPPO then GRPO three-axis reward) recipe producing AgentForesight-7B.

Problem

Existing multi-agent failure analysis is post-hoc, identifying the decisive error only after a trajectory has failed, leaving no window to intervene while the trajectory unfolds.

Approach

Reframes failure analysis as step-by-step online auditing on prefixes only, and trains a 7B auditor in two stages: a boundary-pair preference stage that learns a failure-boundary prior, followed by GRPO with a three-axis (format, step gaussian, agent) reward.

Key results

• AgentForesight-7B reaches 66.44 overall Exact-F1 on AFTraj-2K vs 46.56 for the strongest proprietary baseline DeepSeek-V4-Pro, a +19.9 point gain.
• Overall ASS drops from 1.77 (DeepSeek-V4-Pro) to 0.59 on AFTraj-2K, about 3x lower step localization error.
• On Who&When, AgentForesight-7B reaches 57.69 Step-Acc vs 38.10 for GPT-4.1 under the online protocol.

Surprising findings

• AgentDebug-7B, the post-hoc attributor with full-trajectory hindsight, scores lowest (9.63 overall Exact-F1) under the online protocol.
• A single-stage policy-gradient training collapses to predicting Safe on every prefix, motivating the two-stage recipe.

Limitations

• AFTraj-2K is relatively small (about 2.3K trajectories) and built with GPT-5.4-mini as backbone and GPT-5.4 judges, so labels depend on proprietary models.
• The step-level labels for natural failures rely on LLM proposer/verifier consensus rather than human annotation.
• Evaluated on only three domains and one external benchmark.

Experimental setup

Datasets: AFTraj-2K MATH-500 HumanEval+ MBPP+ GAIA HotpotQA Who&When

Benchmarks: AFTraj-2K Who&When

Metrics: Exact-F1 ASS Step-Acc Agent-Acc false alarm rate

Baselines: GPT-4.1 DeepSeek-V4-Pro Gemini-3-Flash Claude-Haiku-4.5 DeepSeek-V4-Flash Qwen3-32B Qwen3-8B Qwen2.5-7B-Instruct AgentDebug-7B Reflexion-7B ToT-7B Perplexity-7B

Builds on: GRPO DPO AgenTracer AgentDebug Who&When

Closest papers by meaning

• Automata from Agent Traces: Failure and Next-Step Prediction (NeurIPS 2026 · similarity 0.91)
• PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents (NeurIPS 2026 · similarity 0.89)
• AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems? (ICLR 2026 · similarity 0.89)
• CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems (ICML 2026 · similarity 0.89)
• AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators (NeurIPS 2026 · similarity 0.88)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems (ICLR 2026 · similarity 0.87)