Research map / Architectures and efficiency / Mixture of experts
Inner-layer Token Self-modulation as Another Scaling Axis for LLMs
Yebin Yang, Huaijin Wu, Jingtao Han, Yu Wang, Xiaohan Qin, Jingzhi Wang, Debing Zhang, Junchi Yan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add ReToken modulation | Dense-XL 1.5B · Overall average (14 tasks) | accuracy: 22.22 -> 26.54 (+4.32) | Improved |
| add ReToken modulation | Dense-XL 1.5B · MMLU | accuracy: +4.55 | Improved |
| add ReToken modulation | Dense-XL 1.5B · TriviaQA | accuracy: +9.50 | Improved |
| add MoRT | 1.5B-A250M MoE · Overall average (14 tasks) | accuracy: 18.87 -> 22.78 (+3.91) | Improved |
| add MoRT | 3.2B-A0.5B MoE · Overall average (14 tasks) | accuracy: 26.75 -> 32.34 (+5.59) | Improved |
| add MoRT | 3.2B-A0.5B MoE · ARC-C | accuracy: +7.25 | Improved |
| add MoRT | 3.2B-A0.5B MoE · GSM8K | accuracy: +6.31 | Improved |
| use MoRT (routed pool) instead of ReToken | 3.2B-A0.5B MoE · Overall average (14 tasks) | accuracy: 29.24 -> 32.34 | Improved |
| add MoRT at larger backbone scales | MoE 2.3B-A0.5B, 3.9B-A0.6B, A9.8B-A1.4B · Validation loss | loss: reduces final loss by 0.059, 0.064, 0.051 | Improved |
| add MoRT in isoFLOPs analysis | MoE · Compute-optimal loss | loss: 2.27% lower loss; 35% compute saving | Improved |
| double the MoRT parameter count (eta sweep 30 to 130) (ablation) | MoRT on 3.9B-A0.6B MoE · Validation loss | loss: reduces loss by about 0.0118 | Improved |
| add embedding parallelism and token deduplication to MoRT (ablation) | MoRT on 3.2B-A0.5B MoE · Training throughput | tokens/s: 2,749K -> 4,510K | Improved |
Closest papers by meaning
- SpeedLoader: An I/O efficient scheme for heterogeneous and distributed LLM operation (NeurIPS 2024 · similarity 0.87)
- TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (ICLR 2025 · similarity 0.86)
- S$^{2}$FT: Efficient, Scalable and Generalizable LLM Fine-tuning by Structured Sparsity (NeurIPS 2024 · similarity 0.86)
- TokenRouter: Efficient Serving System for Token-Level LLM Routing (NeurIPS 2026 · similarity 0.85)
- SpecEdge: Scalable Edge-Assisted Serving Framework for Interactive LLMs (NeurIPS 2025 · similarity 0.85)
- Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
