Research map / Large language models / Reasoning and chain of thought

Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systems

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Heterogeneous Swarms, which alternates PSO over continuous adjacency matrices (decoded into LLM DAGs for roles) and PSO over LLM weights guided by a JFK-score contribution measure, to jointly optimize roles and weights of a multi-LLM system.

Problem

Existing multi-LLM systems use fixed weights or fixed roles (hand-crafted prompts/workflows), so they cannot flexibly adapt to diverse tasks; the paper jointly optimizes which LLM plays which role in a DAG and the weights of the LLMs.

Approach

Model roles are learned as a DAG of input-output relations by PSO over continuous adjacency matrices decoded with G-Decode; model weights are optimized by PSO using JFK-scores, which estimate each LLM's contribution by averaging system utility over random assignments weighted by how often the LLM appears. The two steps alternate.

Key results

• Heterogeneous Swarms achieves best performance on 11 of 12 datasets, outperforming the second-best baseline by 18.5% on average (e.g., MMLU-pro 0.312 vs 0.254 for Model Swarms; GSM8k 0.481 vs 0.459).
• Collaborative gain of 0.143, 0.184, 0.101, and 0.426 on four datasets, all positive, averaging 0.213.
• Scaling from 2 to 10 LLM experts yields a 27.1% average improvement; a 3x gemma-2-9b system is reported to outperform gemma-2-27b in the appendix comparison.

Measured effects reported in the paper

Surprising findings

• Collaborations of weaker models (bottom half of the pool) can outperform the top-1 individual LLM.
• Role-step and weight-step importance varies by task: weight matters more for knowledge tasks while role matters more for agent tasks.

Limitations

• Requires a pool of LLMs sharing the same architecture so weights live in a common parameter space; black-box or heterogeneous-architecture models are not supported for the weight-step.
• Optimization and inference cost scale with the number of LLMs called, making it expensive.
• Only DAG structures without cycles; limited to text-based node interactions and evaluated only on neural language model nodes.

Experimental setup

Datasets: MMLU-pro Knowledge Crosswords COM2 GSM8k NLGraph Normad GAIA-text AgentBench-KG AgentBench-LTP Qasper AbstainQA WoW

Benchmarks: MMLU-pro Knowledge Crosswords COM2 GSM8k NLGraph Normad GAIA-text AgentBench-KG AgentBench-LTP Qasper AbstainQA WoW

Metrics: accuracy exact match collaborative gain inference speedup

Baselines: Best Single Prediction Merge Data Merge Uniform Soup Dare-Ties Greedy Soup Pack of LLMs LoraHub Model Swarms Chain Star GPT-Swarm

Builds on: particle swarm optimization GPT-Swarm Model Swarms Dare-Ties LoraHub

Closest papers by meaning

• Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligence (ICML 2025 · similarity 0.91)
• HieraMAS: Optimizing Intra-Node LLM Mixtures and Inter-Node Topology for Multi-Agent Systems (ICML 2026 · similarity 0.88)
• Graph-of-Agents: A Graph-based Framework for Multi-Agent LLM Collaboration (ICLR 2026 · similarity 0.88)
• MOC: Multi-Order Communication in LLM-based Multi-Agent Systems (ICML 2026 · similarity 0.86)
• OMAC: A Holistic Optimization Framework for LLM-Based Multi-Agent Collaboration (ICML 2026 · similarity 0.86)
• MAS$^2$: Self-Generative, Self-Configuring, Self-Rectifying Multi-Agent Systems (ICLR 2026 · similarity 0.86)