Research map / Architectures and efficiency / Mixture of experts
DND: Boosting Large Language Models with Dynamic Nested Depth
Tieyuan Chen, Xiaodong Chen, Haoxing Chen, Zhenzhong Lan, Weiyao Lin, Jianguo Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add DND post-training to SFT | Qwen3-1.7B · Dense SFT average (11 benchmarks) | average score: 59.53 -> 61.41 | Improved |
| add DND post-training to SFT | Llama3.2-1B · Dense SFT average (11 benchmarks) | average score: 45.37 -> 47.98 | Improved |
| add DND post-training to SFT | Gemma3-1B · Dense SFT average (11 benchmarks) | average score: 47.08 -> 49.58 | Improved |
| add DND post-training to SFT | Qwen3-30B-A3B · MoE SFT average (17 benchmarks) | average score: 75.70 -> 76.57 | Improved |
| use ITT token selection instead of DND under same compute | Qwen3-1.7B · Dense SFT average (11 benchmarks) | average score: 59.53 -> 59.58 | No clear effect |
| add router control loss and threshold control to DND (ablation) | DND on Qwen3-1.7B · Dense SFT average (11 benchmarks) | average score: 60.54 -> 61.41 | Improved |
| add threshold control on top of router control (ablation) | DND on Qwen3-1.7B · Dense SFT average (11 benchmarks) | average score: 60.68 -> 61.41 | Improved |
| add router control loss on top of threshold control (ablation) | DND on Qwen3-1.7B · Dense SFT average (11 benchmarks) | average score: 60.58 -> 61.41 | Improved |
| lower target selection ratio from 20% to 10% (ablation) | DND on Qwen3-1.7B · Dense SFT average (11 benchmarks) | average score: 61.41 -> 60.33 | Hurt |
| raise target selection ratio from 20% to 30% (ablation) | DND on Qwen3-1.7B · Dense SFT average (11 benchmarks) | average score: 61.41 -> 61.03 | No clear effect |
| apply DND to layers 3:24 instead of 4:23 (ablation) | DND on Qwen3-1.7B · Dense SFT average (11 benchmarks) | average score: 61.41 -> 60.36 | Hurt |
| apply DND to layers 5:22 instead of 4:23 (ablation) | DND on Qwen3-1.7B · Dense SFT average (11 benchmarks) | average score: 61.41 -> 61.05 | Hurt |
Closest papers by meaning
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.89)
- Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA (ICLR 2025 · similarity 0.88)
- Dr.LLM: Dynamic Layer Routing in LLMs (ICLR 2026 · similarity 0.88)
- ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization (NeurIPS 2024 · similarity 0.87)
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.87)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
