Research map / Multi-agent systems and games / LLM agents
SafeSieve: From Heuristics to Experience in Progressive Pruning for LLM-based Multi-Agent Communication
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SafeSieve, a progressive multi-agent communication pruning method that blends LLM-based semantic compatibility scores with accumulated historical edge contribution and uses 0-extension clustering to prune edges, reporting higher accuracy and lower token use than AgentPrune and AgentDropout.
Problem
LLM-based multi-agent systems incur redundant communication and high token overhead, and existing pruning methods either rely on pre-trained GNNs/static design or greedy top-k edge removal, without unifying pre- and post-task optimization.
Approach
Two-stage edge scoring that starts from LLM-assessed semantic compatibility and gradually shifts weight to historical task-correct contribution via time-varying weights and thresholds, with 0-extension clustering (instead of top-k) deciding which cross-cluster edges to prune.
Key results
- SafeSieve reaches 94.01% average accuracy on six benchmarks with DeepSeek-V3, versus 92.78% for AgentPrune and 92.62% for AgentDropout.
- Token usage is reduced by 12.4%-27.8% relative to peer methods; on HumanEval, SafeSieve reaches 95.01% accuracy with 27.8% fewer tokens than the full-connected graph.
- Under malicious-agent injection, SafeSieve drops 1.23 points on average versus 4.59 for AgentPrune; in heterogeneous settings, total cost falls 13.3% relative to AgentPrune (115 to 99.3 cents).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add historical feedback to heuristic cluster pruning (full SafeSieve vs. No-Historic Cluster Prune) (ablation) | SafeSieve on HumanEval · HumanEval | accuracy: +1.23 points (95.01% vs 93.78%) | Helped |
| Add LLM-based heuristic initialization of edge scores (full vs. No-Heuristic Cluster Prune) (ablation) | SafeSieve on HumanEval · HumanEval | accuracy: +0.60 points (95.01% vs 94.41%) | Helped |
| Replace 0-extension clustering with Top-k pruning (ablation) | SafeSieve on HumanEval · HumanEval | accuracy: -1.88 points (93.13% vs 95.01%) | Hurt |
| Post-prune with SafeSieve (progressive heuristic-to-history pruning and 0-extension clustering) vs. full-connected baseline | deepseek-V3 multi-agent system · HumanEval | token usage reduction: 27.8% fewer tokens with 95.01% vs 95.50% accuracy | Helped |
| SafeSieve pruning vs. AgentPrune and AgentDropout on DeepSeek-V3 average accuracy | deepseek-V3 multi-agent system · 6 benchmarks average | average accuracy: 94.01% vs 92.78% (AgentPrune) and 92.62% (AgentDropout) | Helped |
| SafeSieve pruning under injected malicious agents | deepseek-V3 multi-agent system · MMLU, SVAMP, HumanEval | average accuracy drop: 1.23 drop vs 4.59 (AgentPrune) and 2.21 (AgentDropout) | Helped |
| SafeSieve pruning in heterogeneous multi-model collaboration vs. AgentPrune | heterogeneous 5-model team (DeepSeek-V3 commander) · MMLU, SVAMP, HumanEval total cost | total cost: ¢99.31 vs ¢114.59 (-13.3%) | Helped |
| SafeSieve pruning in heterogeneous setting on SVAMP vs. homogeneous SafeSieve | heterogeneous 5-model team · SVAMP | accuracy: +0.17 points (96.77% vs 96.60%) on SVAMP; -0.97 on MMLU (91.42% vs 92.39%); -1.23 on HumanEval (93.78% vs 95.01%) | Mixed |
| Pruning with SafeSieve on GPT-4o-mini vs. AgentPrune | gpt-4o-mini multi-agent system · HumanEval | accuracy: 88.20% vs 86.25% for AgentPrune is higher; but vs GPTSwarm 89.32% is lower | Hurt |
Surprising findings
- Heterogeneous mixing did not improve MMLU accuracy (91.42% vs 92.39% homogeneous), which the authors attribute to a 'barrel effect' from weaker models.
Limitations
- Robustness evaluation uses only injected low-quality agents on three benchmarks, so generality of the defense is unclear.
- Several reported numbers mix sources (e.g., baseline results adapted from AgentDropout and AGP) and are evaluated under different settings, complicating direct comparison.
- The paper relies on an expert LLM (DeepSeek-V3) for semantic compatibility scoring and as evaluator, adding dependence on a strong model.
Experimental setup
Datasets: MMLU GSM8K SVAMP HumanEval AQuA MATH-500
Benchmarks: MMLU GSM8K SVAMP HumanEval AQuA MATH-500
Metrics: accuracy token usage deployment cost accuracy drop under prompt injection
Baselines: Vanilla CoT GPTSwarm G-Designer AgentPrune AgentDropout Top-k pruning
Builds on: AgentPrune AgentDropout GPTSwarm G-Designer
Closest papers by meaning
- Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems (ICLR 2025 · similarity 0.91)
- Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration (ICLR 2026 · similarity 0.88)
- ResMAS: Resilience Optimization in LLM-based Multi-agent Systems (AAAI 2026 · similarity 0.88)
- When Embedding-Based Defenses Fail: Rethinking Safety in LLM-Based Multi-Agent Systems (ICML 2026 · similarity 0.88)
- GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph Modeling (NeurIPS 2025 · similarity 0.88)
- HieraMAS: Optimizing Intra-Node LLM Mixtures and Inter-Node Topology for Multi-Agent Systems (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.