Research map / Multi-agent systems and games / Human-AI and goal-conditioned
KABB: Knowledge-Aware Bayesian Bandits for Dynamic Expert Coordination in Multi-Agent Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KABB, a knowledge-aware Bayesian bandit (Thompson sampling) router that selects top expert LLMs per query for multi-agent aggregation, reporting strong AlpacaEval 2.0 and MT-Bench results at lower cost than MoA.
Problem
Multi-agent LLM systems like Mixture-of-Agents are costly and redundant, and their expert routing relies on static knowledge assumptions that ignore semantic task-expert relations and changing expert capabilities.
Approach
Combines a knowledge distance metric (concept overlap, dependency paths, historical success, team synergy) with time-decayed Beta posteriors and a knowledge-aware Thompson sampling rule to pick a small expert subset per query, followed by an aggregator LLM.
Key results
- KABB reaches 77.9% LC win rate on AlpacaEval 2.0, versus 68.1% for MoA under the same configuration, using 2 experts versus MoA's 6 proposers.
- KABB scores 9.65 average on MT-Bench, and KABB w/o Deepseek (9.47) exceeds GPT-4 Turbo (9.31).
- Deepseek-R1 (80.1% LC win rate) remains higher than KABB on AlpacaEval 2.0, and KABB scores 74.8 on Arena-Hard, below GPT-4 Turbo (82.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use knowledge-aware routing with MAB instead of classifier-based routing with MAB (ablation) | KABB w/o Deepseek · AlpacaEval 2.0 | LC win rate: 60.9 -> 62.4 | Helped |
| use MAB instead of A2C (ablation) | KABB w/o Deepseek with knowledge-aware routing · AlpacaEval 2.0 | LC win rate: 60.2 -> 62.4 | Helped |
| use MAB instead of PPO (ablation) | KABB w/o Deepseek with knowledge-aware routing · AlpacaEval 2.0 | LC win rate: 57.3 -> 62.4 | Helped |
| use MAB instead of MCTS (ablation) | KABB w/o Deepseek with knowledge-aware routing · AlpacaEval 2.0 | LC win rate: 54.8 -> 62.4 | Helped |
| add Deepseek-V3 and Deepseek-R1 to the system (ablation) | KABB · AlpacaEval 2.0 | LC win rate: 62.4 -> 77.9 | Helped |
| add Deepseek models to the system (ablation) | KABB · Arena-Hard | win rate (%): 62.8 -> 74.8 | Helped |
| use multi-agent collaboration and specialization instead of a single LLaMa-3-70B-Instruct (ablation) | KABB-Single-LLaMa3 · AlpacaEval 2.0 | LC win rate: 34.4 -> 34.7 | Helped |
| use multi-agent collaboration and specialization instead of a single LLaMa-3-70B-Instruct (ablation) | KABB-Single-LLaMa3 · Arena-Hard | win rate (%): 46.6 -> 51.4 | Helped |
| set the knowledge distance threshold to 0.75 rather than 0.55 (ablation) | KABB · BBH | Reasoning accuracy (%): 74.8 -> 94.9 | Helped |
| set the knowledge distance threshold to 0.75 rather than 0.95 (ablation) | KABB · BBH | Reasoning accuracy (%): 82.7 -> 94.9 | Helped |
| set the time decay factor to 0.6 rather than 0.2 (ablation) | KABB · BBH | Reasoning accuracy (%): 78.3 -> 94.9 | Helped |
| set the time decay factor to 0.6 rather than 1.0 (ablation) | KABB · BBH | Reasoning accuracy (%): 86.5 -> 94.9 | Helped |
Surprising findings
- KABB underperforms GPT-4 models and Deepseek-R1 on Arena-Hard, and MoA performs similarly to KABB there, suggesting multi-agent setups can add redundancy on closed-form problems.
- The reported ablation numbers are inconsistent: the Table 1 LC win rate for KABB w/o Deepseek (62.4) is reported as identical to the routing ablation KA (MAB) value, and parameter sensitivity tables show identical best values across different parameters.
Limitations
- Theoretical regret bound is stated only as an existence claim with unspecified parameters and lacks a rigorous proof.
- Relies on hand-defined concept sets and expert specialization prompts; the knowledge graph construction is not detailed.
- Lags on conciseness and on closed-form reasoning benchmarks such as Arena-Hard relative to GPT-4 models.
Experimental setup
Datasets: AlpacaEval 2.0 MT-Bench FLASK-Hard Arena-Hard BBH MATH
Benchmarks: AlpacaEval 2.0 MT-Bench FLASK-Hard Arena-Hard BBH MATH
Metrics: length-controlled win rate win rate MT-Bench score Arena-Hard win rate accuracy RAS PWRS TFLOPS
Baselines: Mixture-of-Agents GPT-4 Turbo GPT-4 Omni Deepseek-R1 Deepseek-V3 PPO MCTS A2C Sentence-BERT classifier routing
Builds on: Mixture-of-Agents Thompson Sampling FrugalGPT
Closest papers by meaning
- From Debate to Equilibrium: Belief‑Driven Multi‑Agent LLM Reasoning via Bayesian Nash Equilibrium (ICML 2025 · similarity 0.86)
- LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination (ICML 2025 · similarity 0.86)
- Sequential Probabilistic Uncertainty Estimation for Parallel Multi-Agent Reasoning Systems (NeurIPS 2026 · similarity 0.86)
- Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning (ICLR 2026 · similarity 0.85)
- Automated Composition of Agents: A Knapsack Approach for Agentic Component Selection (NeurIPS 2025 · similarity 0.85)
- Dynamic Knowledge Injection for AIXI Agents (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.