Research map / Architectures and efficiency / Attention and transformers
DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DenseFormer, which adds a learned depth-weighted-average (DWA) module after each transformer block so each block sees a weighted mix of all earlier block outputs, improving perplexity at negligible parameter cost.
Problem
Transformers show diminishing returns from added depth, and deeper models are costly in parameters, memory, and inference time; the paper asks whether better inter-block information flow can match deeper models more efficiently.
Approach
After each block, a DWA module computes a learned scalar-weighted average of the current block output, all previous block outputs, and the input embedding; initialized as identity so training starts from a standard Transformer. Dilation and DWA periodicity sparsify the weights to cut overhead.
Key results
- A 48-block DenseFormer reaches perplexity 17.84 vs 18.61 for a 48-block Transformer (same params), matching a 72-block Transformer (17.82) while being about 1.6x faster at inference (BPS 4.65 vs 4.08 for 72-block baseline... see table).
- A 4x5-DenseFormer (48 blocks) reaches 17.87 perplexity at 5.72 BPS, matching the 64-block Transformer's speed-matched baseline range with better perplexity than the 64-block Transformer (17.94).
- Same-training-time comparison: 4x5-DenseFormer reaches 17.87 perplexity in 8.04h vs 18.33 for a standard Transformer trained 41.5k steps in 8.09h.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DWA after every block | 48-block Transformer · OpenWebText2 | Perplexity: 18.61 (0.02) -> 17.84 (0.00) | Helped |
| Add learned scaling factors to skip connections | 48-block Transformer · OpenWebText2 | Perplexity: 18.61 (0.02) -> 18.45 (0.03) | No clear effect |
| Use DWA dilation 4 (ablation) | 48-block DenseFormer · OpenWebText2 | Perplexity: 17.84 (0.00) -> 17.86 (0.02) | No clear effect |
| Use DWA dilation 4 (ablation) | 48-block DenseFormer · OpenWebText2 | Inference BPS: 4.65 (0.00) -> 5.31 (0.01) | Helped |
| Use DWA dilation 4 and period 5 (ablation) | 48-block DenseFormer · OpenWebText2 | Perplexity: 17.84 (0.00) -> 17.87 (0.02) | No clear effect |
| Use DWA dilation 4 and period 5 (ablation) | 48-block DenseFormer · OpenWebText2 | Inference BPS: 4.65 (0.00) -> 5.72 (0.00) | Helped |
| Add DWA after every block | 24-block Transformer · PG-19 | Perplexity: 20.13 -> 19.60 | Helped |
| Add DWA after every block | 48-block Transformer · OpenWebText2, 512-token sequences | Perplexity: 18.28 ± 0.03 -> 17.73 ± 0.02 | Helped |
| Use DWA with dilation 4 and period 5 | 48-block Transformer · OpenWebText2, same training time | Perplexity: 18.33 (0.00) -> 17.87 (0.02) | Helped |
| Restrict DWA access to the last 4 blocks and embeddings (ablation) | DenseFormer · OpenWebText2, alternative sparsity patterns | Perplexity: 17.87 (0.02) -> 18.28 (0.01) | Hurt |
| Use a single DWA after the last block (ablation) | DenseFormer · OpenWebText2, alternative sparsity patterns | Perplexity: 17.87 (0.02) -> 18.33 (0.02) | Hurt |
| Start training DWA weights after 1k iterations instead of at initialization (ablation) | 4×5-DenseFormer · OpenWebText2 | Perplexity: 17.87 (0.02) -> 17.99 | Hurt |
Surprising findings
- Simply adding learned scaling to skip connections (Skips With Gains) gives little benefit, suggesting the gain comes from direct access to all previous block outputs.
- Dropping more than about 15% of the smallest DWA weights at inference sharply increases perplexity despite the weights being small.
Limitations
- Naive implementation is slow without dilation/periodicity; efficient sharding across nodes for large-scale distributed training is left open.
- Most experiments use models up to 90 blocks and ~17B tokens of OpenWebText2; scale beyond this is not tested.
Experimental setup
Datasets: OpenWebText2 PG-19
Benchmarks: OpenWebText2 PG-19
Metrics: perplexity inference batches per second training time parameter count
Baselines: Standard Transformer (same depth) Skips With Gains Last K sparsity Connect to Last Depth-wise Attention
Builds on: DenseNet Residual connections RoPE AdamW Depth-wise Attention
Closest papers by meaning
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.88)
- MemoryFormer : Minimize Transformer Computation by Removing Fully-Connected Layers (NeurIPS 2024 · similarity 0.87)
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.87)
- SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention (NeurIPS 2024 · similarity 0.87)
- The Curse of Depth in Large Language Models (NeurIPS 2025 · similarity 0.87)
- Transformer Layers as Painters (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.