Research map / Architectures and efficiency / Mixture of experts
BAM! Just Like That: Simple and Efficient Parameter Upcycling for Mixture of Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BAM (Branch-Attend-Mix), which upcycles specialized dense experts' attention as well as FFN parameters into soft-routed mixture-of-attention experts plus FFN experts, using a parallel attention transformer, and outperforms BTX on perplexity and downstream tasks at 590M and 2B seed scales.
Problem
Upcycling multiple specialized dense models into an MoE (as in BTX) only reuses FFN parameters and averages attention parameters, which wastes specialized attention knowledge and can degrade performance.
Approach
Initialize both attention experts (soft-routed Mixture of Attention, with per-expert or shared KV projections) and FFN experts directly from the specialized dense models instead of averaging attention, and use a parallel attention transformer so attention and FFN experts compute concurrently.
Key results
- At 2B scale, BAM (Expert KV, data-matched) reaches 34.02% average downstream score vs. 32.71% for BTX.
- At 590M scale, BAM (Shared KV, data-matched) reaches 10.32 average perplexity vs. 10.72 for BTX (lower is better); BAM Expert KV DM reaches 9.70.
- BAM is slower at inference than BTX (6.17s vs 4.81s for 16 tokens), while the KV-sharing variant reduces latency to 5.96s.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| upcycle attention parameters into experts, including expert KV | BAM · Small-scale perplexity, Average | perplexity: 10.72 -> 9.70 | Helped |
| upcycle attention parameters into experts, including expert KV | BAM · Large-scale perplexity, Average | perplexity: 3.27 -> 3.19 | Helped |
| upcycle attention parameters into experts, including expert KV | BAM · Large-scale downstream tasks, Average | score: 32.71% -> 34.02% | Helped |
| replace soft attention routing with top-2 routing (ablation) | BAM MoA · Small-scale perplexity ablation, Average | perplexity: 10.47 -> 10.72 | Hurt |
| replace soft attention routing with top-1 routing (ablation) | BAM MoA · Small-scale perplexity ablation, Average | perplexity: 10.47 -> 10.81 | Hurt |
| match BAM's total parameters by increasing BTX from 4 to 6 FFN experts (ablation) | BTX · Small-scale perplexity, Code | perplexity: 3.44 -> 3.99 | Hurt |
| match BAM's total and active parameters with BTX 6 experts and top-3 routing (ablation) | BTX · Small-scale perplexity, Code | perplexity: 3.44 -> 3.55 | Hurt |
| share key-value parameters and KV cache across attention experts | BAM · Inference latency, 16 generated tokens | latency: 6.17s -> 5.96s | Helped |
| use BAM rather than standard BTX | BAM · Inference latency, 16 generated tokens | latency: 4.81s -> 6.17s | Hurt |
Surprising findings
- Top-1 and top-2 sparse routing for attention experts do not beat BTX; only soft routing consistently outperforms the baseline.
- Simply matching BTX's total or active parameter count (more FFN experts or top-3 routing) does not close the gap to BAM.
Limitations
- Inference latency remains higher than the BTX baseline (6.17s vs 4.81s), attributed to per-expert KV caches.
- Soft routing over all attention experts increases attention FLOPs compared to sparse MoE.
- Experiments use only four experts (seed plus code, math, law) and in-house seed models; results at larger scale are not shown.
Experimental setup
Datasets: MathGLM GSM8K proof-pile-2 MathPile StarCoder pile-of-law HUPD Common Crawl
Benchmarks: GSM8K MATH HumanEval MBPP LegalBench Natural Questions TriviaQA ARC-Easy ARC-Challenge SIQA PIQA WinoGrande
Metrics: perplexity accuracy
Baselines: BTX (Branch-Train-MiX) dense seed model dense expert models
Builds on: Branch-Train-MiX Mixture of Attention Sparse Upcycling parallel attention transformer switch transformer load balancing
Closest papers by meaning
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.93)
- DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Models (CVPR 2025 · similarity 0.91)
- Advancing Expert Specialization for Better MoE (NeurIPS 2025 · similarity 0.91)
- Scaling Laws for Fine-Grained Mixture of Experts (ICML 2024 · similarity 0.91)
- MoEQuant: Enhancing Quantization for Mixture-of-Experts Large Language Models via Expert-Balanced Sampling and Affinity Guidance (ICML 2025 · similarity 0.90)
- Scaling Laws for Upcycling Mixture-of-Experts Language Models (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.