Research map / Multi-agent systems and games / LLM agents
MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills multi-round multi-agent discussions among GPT-4, Bard, and Claude2 (ReConcile) into a small student (Mistral-7B-Instruct) using a GCN over interaction graphs plus next-token, contrastive, and node-classification losses.
Problem
Multi-agent LLM discussions improve reasoning but are very expensive at inference and do not yield a single standalone model; the paper asks how to distill multi-teacher interaction structure into a small efficient student.
Approach
Represent multi-agent discussions as multi-agent interaction graphs (MAGs) with correctness-labeled nodes and round-conditioned edges, then distill them into a student augmented with a GCN using three objectives: next-token prediction on correct chains, a margin contrastive loss between correct and incorrect chains, and a node-correctness classification loss over the graph structure. The GCN is di
Key results
- MAGDi on Mistral-7B-Instruct averages 56.88% across five benchmarks vs 46.17% zero-shot and 52.27% for the best single-teacher SiT-GPT4 (+4.61 points over SiT-GPT4).
- MAGDi reduces output tokens versus ReConcile by 4.5x on average (970.4 to 216.9), up to 9x on StrategyQA.
- On GSM8K with self-consistency over 10 responses, MAGDi reaches 67.42% vs 58.62% for SiT-GPT4 and 48.44% for the base model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill from all nodes and interaction edges instead of only GPT-4 positive samples | Mistral-7B-Instruct · Five benchmarks, average accuracy | accuracy: 52.27 -> 56.88 | Helped |
| Use correct reasoning from multiple teachers rather than only round-0 correct reasoning (ablation) | MAGDi · Five benchmarks, average accuracy | accuracy: + 1.50% | Helped |
| Add correct post-interaction reasoning from multiple teachers (ablation) | MAGDi-R0 · Five benchmarks, average accuracy | accuracy: 53.77 -> 54.81 | Helped |
| Add incorrect reasoning and a contrastive objective (ablation) | MAGDi-CN · Five benchmarks, average accuracy | accuracy: 54.81 -> 55.47 | Helped |
| Add graph structure and interaction objective to all-node distillation (ablation) | MAGDi-AN · Five benchmarks, average accuracy | accuracy: 55.47 -> 56.88 | Helped |
| Use structured multi-teacher distillation rather than DSS-MT on all nodes | Mistral-7B-Instruct · Five reasoning benchmarks | accuracy: 54.76 -> 56.88 | Helped |
| Use GCN rather than token-level attention to model interactions (ablation) | MAGDi · StrategyQA | accuracy: 68.56 -> 74.24 | Helped |
| Use GCN rather than node-level attention to model interactions (ablation) | MAGDi · StrategyQA | accuracy: 69.87 -> 74.24 | Helped |
| Remove G3 interaction graphs from training (ablation) | MAGDi · CSQA | accuracy: 72.56 -> 70.65 | Hurt |
| Also remove G2 interaction graphs from training (ablation) | MAGDi without G3 · CSQA | accuracy: 70.65 -> 69.65 | Hurt |
| Replace interaction-pattern edges with fully connected edges (ablation) | MAGDi · CSQA | accuracy: 72.56 -> 71.22 | Hurt |
| Make interaction-pattern edges undirected instead of directed (ablation) | MAGDi · CSQA | accuracy: 72.56 -> 72.61 | No clear effect |
Surprising findings
- The multi-task MAGDi model is within about 1 point of task-specific models (55.12 vs 56.89 average), and it also beats single-teacher distillation on OOD BoolQ and SVAMP.
- A fully-connected graph variant performs worse than interaction-based edges, while edge directionality has marginal effect.
Limitations
- Relies on API-based teacher LLM interactions (GPT-4, Bard, Claude2) to construct training graphs, which are expensive to collect.
- Absolute accuracy on MATH remains low (12.76% for the student vs 41% for ReConcile), and the gap to teachers is substantial on math.
- Efficiency comparison uses output tokens only, not wall-clock or cost, and teacher sizes are unknown.
Experimental setup
Datasets: StrategyQA CommonsenseQA ARC-c GSM8K MATH BoolQ SVAMP
Benchmarks: StrategyQA CommonsenseQA ARC-c GSM8K MATH BoolQ SVAMP
Metrics: accuracy output token count self-consistency accuracy
Baselines: No teacher zero-shot Mistral-7B-Instruct SiT-GPT4 SiT-Claude2 SiT-Bard DSS-MT ReConcile
Builds on: ReConcile Distilling Step-by-Step LoRA self-consistency
Closest papers by meaning
- Distilling LLM Agent into Small Models with Retrieval and Code Tools (NeurIPS 2025 · similarity 0.91)
- Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024 · similarity 0.88)
- Benefits and Limitations of Communication in Multi-Agent Reasoning (ICLR 2026 · similarity 0.88)
- Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberation (ICLR 2026 · similarity 0.88)
- Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration (ICLR 2026 · similarity 0.88)
- Latent Collaboration in Multi-Agent Systems (ICML 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.