Research map / Architectures and efficiency / Mixture of experts

Sparse Layers are Critical to Scaling Looped Language Models

NeurIPS 2026 · Architectures and efficiency › Mixture of experts · approach: Efficient MoE language model systems · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Controlled isoFLOP study of dense vs. MoE transformers with and without layer looping, showing Looped-MoE scales better than dense Base and that loop boundaries enable better training-free early exits.

Problem

Looped language models save parameters but scale worse than standard transformers with unique layers, and their inference cost remains high; it is unclear whether sparse experts fix the looped scaling deficit.

Approach

Replace dense FFNs in looped transformers with top-k MoE layers so routing differs across loop passes, restoring expressivity without extra parameters; looped models also converge earlier at loop boundaries, making them good early-exit points.

Key results

• Looped-MoE (8x2) achieves Core 9 average 39.6 vs. 38.7 for Base at 10^18 FLOPs with fewer stored parameters (216M vs 246M).
• At 10% FLOPs saved via entropy early exit, Looped-MoE 2x8 reaches perplexity 42.0 vs 55.4 for Base and 75.7 for MoE.
• Routing diverges across loops: 25-53% of tokens get fully non-overlapping expert sets between passes in most layers.

Surprising findings

• Non-looped MoE scores lowest on downstream average (36.4) despite the best test loss.
• Looped dense models scale strictly worse than Base, so looping alone is detrimental to scaling.

Limitations

• Largest models are 305M active / 711M stored parameters; scaling claims rely on extrapolation to 1B/7B.
• Early-exit results are theoretical FLOP savings; end-to-end throughput not measured.
• muP transfer does not hold across depth, so depth is held constant.

Experimental setup

Datasets: FineWeb-Edu

Benchmarks: AI2 OLMES Core 9 (ARC-E, ARC-C, BoolQ, CSQA, HellaSwag, OBQA, PIQA, SIQA, WinoG)

Metrics: test loss perplexity average accuracy JSD convergence expert overlap

Baselines: Base dense transformer Looped dense MoE (non-looped) MoEUT Ouro

Builds on: Mixtral Universal Transformers Maximal Update Parameterization LayerSkip

Closest papers by meaning

• Looped Diffusion Language Models (NeurIPS 2026 · similarity 0.90)
• LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation (ICLR 2026 · similarity 0.89)
• Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models (ICML 2025 · similarity 0.89)
• Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA (ICLR 2025 · similarity 0.89)
• The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.88)
• SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention (NeurIPS 2024 · similarity 0.88)