Research map / Architectures and efficiency / Mixture of experts

Swimba: Switch Mamba Model Scales State Space Models

NeurIPS 2026 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Swimba, an MoE-parameterized Mamba-2 SSM layer that mixes expert-produced SSM streams in parameter space to keep a single recurrence, and adapts Nemotron-H-8B into a 14B model with near-equal FLOPs.

Problem

Applying MoE to SSM token mixers can multiply the cost of the recurrent state update when each expert keeps its own state trajectory; the paper seeks expert specialization in selective SSMs while preserving the single-recurrence cost.

Approach

Distinguishes two MoE-SSM designs (separated SSMs with per-expert states vs. MoE-parameterized SSM mixing expert B/C/X streams in parameter space with a shared transition A) and builds Swimba on the latter, so the recurrence is evaluated once per token regardless of expert count.

Key results

• Swimba-14B average accuracy 70.81 vs Nemotron-H-8B 69.47 (normalized average 72.34 vs 70.98) across nine LM-Eval-Harness tasks.
• Per-token inference FLOPs 1.51282e10 for Swimba-14B vs 1.51263e10 for Nemotron-H-8B (<0.2% difference).
• vLLM benchmarking shows Swimba-14B has slightly lower throughput and higher latency, with as little as a 10% slowdown on a single 40GB A100.

Surprising findings

• Swimba-14B shows a larger runtime gap than its near-identical FLOPs would suggest, which the authors attribute to routing overhead.
• Swimba-14B underperforms the baseline on BoolQ, OpenbookQA and some normalized metrics, so gains are not uniform.

Limitations

• Gains on benchmarks are small and within reported standard deviations.
• Runtime overhead from routing makes throughput and latency slightly worse than baseline.
• Continued training details (tokens, compute, hyperparameters) are sparse.

Experimental setup

Datasets: Llama Nemotron Post-Training Dataset (SFT portion)

Benchmarks: BoolQ OpenBookQA RTE MMLU PIQA WinoGrande HellaSwag ARC-Challenge ARC-Easy

Metrics: accuracy normalized accuracy throughput latency FLOPs per token

Baselines: Nemotron-H-8B

Builds on: Mamba-2 Nemotron-H Switch Transformer Mixture of Experts

Closest papers by meaning

• Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection (NeurIPS 2025 · similarity 0.89)
• Demystifying the Token Dynamics of Deep Selective State Space Models (ICLR 2025 · similarity 0.88)
• Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast (NeurIPS 2024 · similarity 0.85)
• MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.85)
• MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems (NeurIPS 2025 · similarity 0.85)
• MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design (ICML 2025 · similarity 0.85)