Research map / Architectures and efficiency / Pruning and efficient tuning
CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structure
Boao Kong, Junzhu Liang, Yuxi Liu, Renjia Deng, Kun Yuan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use a full-rank first layer instead of low-rank (ablation) | CR-Net · LLaMA-2 350M C4 | validation perplexity: 19.68 -> 18.95 | Improved |
| make the scaling factor beta learnable instead of fixed (ablation) | CR-Net · LLaMA-2 350M C4 | training stability | Improved |
| use higher ranks in middle layers and lower in side layers (ablation) | CR-Net · LLaMA-2 350M C4 | validation perplexity | Improved |
| use previous-layer residual instead of learnable first-layer skip (Learnable-ResFormer) or DenseFormer (ablation) | CR-Net · LLaMA-2 1B C4 | validation perplexity | Improved |
| replace full-rank training with CR-Net at aligned parameters | CR-Net · LLaMA-2 1B C4 | validation perplexity: 15.56 -> 15.22 | Improved |
| CR-Net vs CoLA at aligned parameters | CR-Net · LLaMA-2 60M C4 | validation perplexity: 34.04 -> 32.76 | Improved |
| CR-Net vs full-rank at 350M | CR-Net · LLaMA-2 350M C4 | validation perplexity: 18.80 -> 18.95 | Hurt |
| CR-Net with re-computation vs CoLA-M | CR-Net · LLaMA-2 7B C4 80K steps | validation perplexity: 13.82 -> 13.72 | Improved |
| CR-Net with re-computation vs 8-bit Adam | CR-Net · LLaMA-2 13B C4 40K steps | validation perplexity: 17.85 -> 18.12 | Hurt |
| use tailored re-computation instead of vanilla gradient checkpointing | CR-Net · LLaMA2-7B activation memory | memory (GB): 51.22 -> 23.35 | Improved |
| CR-Net re-computation vs CoLA-M activation memory | CR-Net · LLaMA2-7B activation memory | memory (GB): 24.78 -> 23.35 | Improved |
| CR-Net at aligned memory vs Apollo | CR-Net · LLaMA-2 1B C4 | validation perplexity: 14.20 -> 14.05 | Improved |
Closest papers by meaning
- GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling (NeurIPS 2025 · similarity 0.89)
- Sparsing Law: Towards Large Language Models with Greater Activation Sparsity (ICML 2025 · similarity 0.88)
- Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LN (ICLR 2025 · similarity 0.88)
- Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenation (NeurIPS 2025 · similarity 0.88)
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.88)
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
