Research map / Multi-agent systems and games / LLM agents

Beyond LLM-Based Reasoning: Lightweight GNNs for Agent Failure Attribution

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: LLM multi-agent decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AFANet, a lightweight GNN over turn-level conversation graphs with TF-IDF/SVD deviation and statistical features, that matches or beats LLM baselines on agent failure attribution at near-zero inference cost.

Problem

Agent failure attribution in LLM-based multi-agent systems: identifying the faulty agent and its error type from a failed trajectory, where LLM-based methods are costly and achieve limited accuracy.

Approach

Replace generative LLM reasoning with a lightweight GNN over a turn-level graph (temporal and same-agent edges) whose node features combine TF-IDF SVD deviation signals, statistical features, and sentence embeddings; agent-level pooling feeds a fault-detection plus error-classification head.

Key results

• AFANet reaches 17.42 micro-F1 pair-level on AEGIS-Bench and 24.82 average across six metrics, competitive with the best LLM baselines (Qwen2.5-14B+S average 26.51 on the table).
• AFANet has 65K trainable parameters with 1.16s in-domain inference versus 199-367s for 7B/14B LLM inference over the test set.
• Test-time entropy minimization on LayerNorm parameters improves the Who&When average from 13.96 to 14.75 (Pair-Faulty variant).

Surprising findings

• Removing all graph edges only modestly degrades performance (22.78 vs 24.82 average), and the GNN-free variant remains close (22.24), suggesting the hand-crafted features carry much of the signal.
• Even the strongest LLM baselines achieve limited accuracy on Who&When, with several models scoring near random on error-level metrics.

Limitations

• Generalization across substantially different multi-agent settings remains challenging under OOD shift.
• Lightweight structural modeling may miss failures requiring deep semantic understanding and long-horizon reasoning.

Experimental setup

Datasets: AEGIS-Bench Who&When

Benchmarks: AEGIS-Bench Who&When

Metrics: micro-F1 macro-F1 agent-level F1 error-level F1 pair-level F1

Baselines: Qwen2.5-7B-Instruct Qwen2.5-14B-Instruct Qwen3-8B GPT-4.1 GPT-4o-mini o3 Gemini-2.5-Flash Gemini-2.5-Pro Claude-Sonnet-4

Builds on: GCN GAT GraphSAGE TENT sentence-BERT

Closest papers by meaning

• AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems? (ICLR 2026 · similarity 0.92)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems (ICLR 2026 · similarity 0.90)
• Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems (ICML 2025 · similarity 0.90)
• CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems (ICML 2026 · similarity 0.89)
• Distilling LLM Agent into Small Models with Retrieval and Code Tools (NeurIPS 2025 · similarity 0.88)
• Who&When Pro: Can LLMs Really Attribute Failures in AI Agents? (NeurIPS 2026 · similarity 0.88)