Research map / Architectures and efficiency / RNNs and normalization
Unified Scaling Laws for Routed Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the separable scaling law with a bilinear interaction between log N and log E (ablation) | Sinkhorn-base · Leave-one-out model predictions | RMSLE: 80e-4 -> 60e-4 | Helped |
| Replace the separable scaling law with a bilinear interaction between log N and log E (ablation) | RLR · Leave-one-out model predictions | RMSLE: 90e-4 -> 57e-4 | Helped |
| Replace the separable scaling law with a bilinear interaction between log N and log E (ablation) | Hash · Leave-one-out model predictions | RMSLE: 90e-4 -> 60e-4 | Helped |
| Add saturation to the expert-count term of the bilinear scaling law (ablation) | Sinkhorn-base · Leave-one-out model predictions | RMSLE: 60e-4 -> 58e-4 | Helped |
| Add saturation to the expert-count term of the bilinear scaling law (ablation) | RLR · Leave-one-out model predictions | RMSLE: 57e-4 -> 56e-4 | Helped |
| Add saturation to the expert-count term of the bilinear scaling law (ablation) | Hash · Leave-one-out model predictions | RMSLE: 60e-4 -> 56e-4 | Helped |
| Use a learned baseline and policy entropy regularization (ablation) | RLR · Routed language models across model sizes and expert counts | validation loss | Helped |
| Use nucleus sampling instead of greedy REINFORCE selection (ablation) | RLR · Routed language models | validation loss | Helped |
| Add a learned baseline to nucleus-sampled REINFORCE (ablation) | RLR · Routed language models | validation loss | Helped |
| Use sampling from the full expert distribution instead of greedy expert selection (ablation) | RLR · Larger-scale routed language models | validation loss | Hurt |
| Add policy entropy regularization together with nucleus sampling (ablation) | RLR · Routed language models | validation loss | No clear effect |
| Add input jitter to activations sent to the router (ablation) | Sinkhorn-base · Routed language models | validation loss | No clear effect |
Closest papers by meaning
- RACER: Risk-Aware Calibrated Efficient Routing for Large Language Models (ICML 2026 · similarity 0.88)
- Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.88)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.88)
- Observational Scaling Laws and the Predictability of Langauge Model Performance (NeurIPS 2024 · similarity 0.87)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.87)
- Lookahead Routing for Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.