Research map / Architectures and efficiency / RNNs and normalization
Scaling depth capacity via zero/one-layer model expansion
Zhiqi Bu
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| initialize new layers randomly or by copying instead of zeros (ablation) | progressive depth expansion · GPT2/LLAMA3/Qwen3/DeepSeekV3/Mixtral | validation loss / compute efficiency: not given | Improved |
| zero-initialize new layers (ablation) | progressive depth expansion · GPT2 1-layer to 12-layer | validation loss / compute efficiency: not given | Hurt |
| copy all layers (stack or interpolate) instead of copying only the last layer (ablation) | multi-layer expansion · GPT2 6-layer, ResNet50 | validation loss / compute efficiency: not given | Improved |
| copying interpolate vs copying stack (ablation) | multi-layer expansion · GPT2 6-layer | validation loss / compute efficiency: almost indistinguishable | No clear effect |
| use WSD schedule instead of cosine schedule (ablation) | zero-layer progressive training · GPT2 and ResNet | validation loss / compute efficiency: small model catches up until tau 0.8T vs 0.5T (GPT) | Improved |
| expand depth later in the stable phase of WSD (ablation) | progressive training · GPT2 1-layer | validation loss / compute efficiency: mixing time about 30k iterations at 10% and 60% horizon | No clear effect |
| expand at 80% of iterations after decay begins (ablation) | one-layer progressive training · GPT2 12-layer | validation loss / compute efficiency: cannot mix well | Hurt |
| use multi-stage expansion (0 to 2 to 12) instead of single-stage (ablation) | progressive training · GPT2 12-layer | validation loss / compute efficiency: similar final loss | No clear effect |
| expand from more than 2 layers instead of zero/one layer (ablation) | progressive training · GPT2 12/24/36-layer | validation loss / compute efficiency: not given | Improved |
| use muP-scaled learning rate with Muon-NSGD (ablation) | progressive training · GPT2 and ResNet | validation loss / compute efficiency: optimal learning rate transfers across sizes | Improved |
| apply zero/one-layer progressive training to MoE | DeepSeekV3 and Mixtral · OpenWebText | validation loss / compute efficiency: 3.4x to 5.1x | Improved |
Closest papers by meaning
- Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks (ICLR 2024 · similarity 0.89)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.87)
- Navigating Scaling Laws: Compute Optimality in Adaptive Model Training (ICML 2024 · similarity 0.87)
- Feature Learning Dynamics in Infinite-Depth Neural Networks (NeurIPS 2026 · similarity 0.86)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.86)
- $\mu$PC: Scaling Predictive Coding to 100+ Layer Networks (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
