Research map / Architectures and efficiency / Pruning and efficient tuning
Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies three structured linear parameterizations (LowRank, BlockShuffle, BlockDense) of Transformer FFNs trained from scratch up to 1.3B parameters, proposes a pre-merge trick for decoding and a self-guided training regime to fix poor optimization dynamics.
Problem
FFNs dominate parameters, FLOPs and latency in LLMs, yet structured (low-rank / block) approximations are mostly studied for small CNNs or fine-tuning, and they train poorly when used from initialization at LLM scale.
Approach
Replace dense FFN projections with structured products U(Vx) trained from scratch, add a pre-merge technique to recover dense-level decoding speed, and use self-guided training that mixes in a dense residual branch with a cosine-decaying weight (stochastically applied) to guide structured factors through early training.
Key results
- Wide and structured LowRank model with 464M params matches the 729M dense Transformer-l's perplexity (14.27 vs 14.29) at equal training FLOPs, with a 17% throughput gain.
- Self-guided training reduces the perplexity gap of structured Transformer-xl FFNs (32% FFN params) from 1.0-1.3 to 0.4-0.6 at equal training FLOPs.
- LowRank with 32% FFN parameters gives a 1.35x whole-model training speedup on Transformer-xl at a perplexity cost of about 1.1 points (13.55 vs 12.46).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace dense FFN layers with LowRank using 32% of FFN parameters | Transformer-xl · RefinedWeb validation | PPL: 12.46 -> 13.55 | Hurt |
| Apply self-guided training to LowRank FFNs | Transformer-m · RefinedWeb validation | PPL: 20.60 -> 19.90 | Helped |
| Apply self-guided training to BlockDense FFNs | Transformer-m · RefinedWeb validation | PPL: 20.85 -> 20.10 | Helped |
| Apply self-guided training to BlockShuffle FFNs | Transformer-m · RefinedWeb validation | PPL: 21.12 -> 20.36 | Helped |
| Place the dense or low-rank projection before the block-diagonal projection (ablation) | BlockDense on Transformer-s · RefinedWeb, 2.2B training tokens | PPL: 29.17 -> 29.65 | Hurt |
| Remove the second shuffle operation (ablation) | BlockShuffle · RefinedWeb, Transformer-s and Transformer-m, 32% FFN parameters | PPL: 29.95 -> 29.89; 21.12 -> 21.19 | No clear effect |
| Use orthonormal instead of random Gaussian initialization (ablation) | BlockShuffle (B=4) · WikiText-103, 4-layer Transformer | PPL: 27.24 -> 25.33 | Helped |
| Use orthonormal instead of random Gaussian initialization (ablation) | BlockDense (B=2, R=128) · WikiText-103, 4-layer Transformer | PPL: 28.25 -> 26.63 | Helped |
| Use Frobenius weight decay instead of standard weight decay (ablation) | Structured FFNs · Language modeling experiments | PPL | No clear effect |
| Directly decompose the dense weight at the midpoint of training (ablation) | LowRank on Transformer-s · RefinedWeb | PPL: 28.02 -> 28.56 | Hurt |
| Progressively decrease rank during training (ablation) | LowRank on Transformer-s · RefinedWeb | PPL: 28.02 -> 29.35 | Hurt |
| Use a slower self-guided training schedule (ablation) | LowRank on Transformer-s · RefinedWeb | PPL: 28.02 -> 27.90 | Helped |
Surprising findings
- Reversing the order of the two projections in BlockDense made perplexity worse (29.17 to 29.65).
- On CIFAR-10, BlockShuffle with a structured first layer in a 5-layer MLP outperformed the dense version (67.08 vs 66.99 with same structure), while structuring the first layer of ViT hurt accuracy.
Limitations
- BlockDense and BlockShuffle have limited hyperparameter exploration
- Focus is language modeling with only limited vision experiments; models not comparable to today's practical LLMs such as LLaMA-3
- Optimal scaling laws for structured matrices were not explored
Experimental setup
Datasets: RefinedWeb WikiText-103 CIFAR-10
Benchmarks: RefinedWeb validation perplexity PIQA HellaSwag Winogrande ARC LAMBADA CIFAR-10
Metrics: perplexity loss accuracy training FLOPs latency throughput
Baselines: Dense Transformer GQA Transformer Direct decomposition Progressive decreasing rank
Builds on: Chinchilla scaling law Monarch matrices ShuffleNet GQA LoRA
Closest papers by meaning
- Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs (ICLR 2024 · similarity 0.90)
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.90)
- Efficient Large Language Model Inference with Neural Block Linearization (NeurIPS 2025 · similarity 0.90)
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.90)
- Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? (NeurIPS 2025 · similarity 0.90)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.