Research map / Architectures and efficiency / Attention and transformers
MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MUDDFormer, which replaces residual connections with multiway dynamic dense connections that generate position-wise cross-layer weights for the query, key, value and residual streams, improving language modeling and ViT accuracy at small overhead.
Problem
Residual connections in deep Transformers create a bottleneck for cross-layer information flow, causing representation collapse and diminishing returns from depth; existing dense connections use static, shared weights.
Approach
Generate dense connection weights dynamically per sequence position from hidden states, and decouple the block input into query, key, value and residual streams, each with its own depth-wise aggregation over all preceding layer outputs (multiway dynamic dense connections).
Key results
- MUDDPythia-2.8B matches Pythia-6.9B in Pile perplexity (6.29 vs 6.29) and downstream average, and reaches 57.0 five-shot average vs Pythia-12B's 57.2 while adding 0.23% parameters and 0.4% FLOPs.
- MUDDFormer-834M matches the loss of Transformer++ trained with about 1.89x compute, with gains that persist across scales from 405M to 1.4B.
- On ImageNet, MUDDViT-S/16 reaches 78.1% top-1 at epoch 300 vs 76.0% for ViT-S/16 at only 1.007x relative size.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MUDD connections to ViT-S/16 | ViT-S/16 · ImageNet-1k validation, epoch 300 | top-1 accuracy: 76.0 -> 78.1 | Helped |
| Add static dense connections (ablation) | Transformer++ · Pile validation, 405M | perplexity: 11.68 -> 11.44 | Helped |
| Add dynamic dense connections (ablation) | Transformer++ with static dense connections · Pile validation, 405M | perplexity: 11.44 -> 11.09 | Helped |
| Use multiway static dense connections (ablation) | Transformer++ with dynamic dense connections · Pile validation, 405M | perplexity: 11.09 -> 11.27 | Hurt |
| Use multiway dynamic dense connections (ablation) | Transformer++ with dynamic dense connections · Pile validation, 405M | perplexity: 11.09 -> 10.83 | Helped |
| Add parameter re-allocation to Transformer++ (ablation) | Transformer++ · Pile validation, 405M | perplexity: 11.68 -> 11.93 | Hurt |
| Add parameter re-allocation to MUDDFormer (ablation) | MUDDFormer · Pile validation, 405M | perplexity: 10.83 -> 10.77 | Helped |
| Replace query-stream dense connections with residual connections (ablation) | MUDDFormer · Pile validation, 405M | perplexity: 10.77 -> 10.89 | Hurt |
| Replace key-stream dense connections with residual connections (ablation) | MUDDFormer · Pile validation, 405M | perplexity: 10.77 -> 10.90 | Hurt |
| Replace value-stream dense connections with residual connections (ablation) | MUDDFormer · Pile validation, 405M | perplexity: 10.77 -> 11.05 | Hurt |
| Replace residual-stream dense connections with residual connections (ablation) | MUDDFormer · Pile validation, 405M | perplexity: 10.77 -> 11.14 | Hurt |
| Use sparse MUDDFormer-2×2 instead of fully dense MUDDFormer-1×1 (ablation) | MUDDFormer · Pile validation, 405M | perplexity: 0.18 increase in ppl | Hurt |
Surprising findings
- MUDD connections gave a larger loss reduction on the MoE model (-0.0641) than on the dense model (-0.0596).
- Parameter re-allocation helped MUDDFormer but deteriorated Transformer++ (ppl 11.93 vs 11.68).
Limitations
- Measured training and inference overheads (roughly 4-20% slowdown, 10-16% lower throughput at 2.8B) exceed theoretical estimates; kernels are not custom-fused.
- Gains on ViT shrink over longer training (many epochs), suggesting diminishing benefit with repeated data.
- Simplifications of the dynamic weight generator (no softmax, no input-dependent keys) are justified empirically rather than theoretically.
Experimental setup
Datasets: The Pile ImageNet-1k FLAN Collection (subset for perplexity)
Benchmarks: LAMBADA PIQA WinoGrande ARC-E ARC-C SciQ LogiQA BoolQ HellaSwag RACE-M RACE-H ImageNet-1k
Metrics: perplexity accuracy validation loss training throughput inference speed
Baselines: Transformer++ DenseFormer Hyper-Connections DDFormer Pythia-1.4B Pythia-2.8B Pythia-6.9B Pythia-12B OLMoE-style MoE ViT-S/16 ViT-M/16
Builds on: DenseNet DenseFormer Hyper-Connections Pythia Transformer++
Closest papers by meaning
- Do Efficient Transformers Really Save Computation? (ICML 2024 · similarity 0.87)
- Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models (NeurIPS 2025 · similarity 0.87)
- SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention (NeurIPS 2024 · similarity 0.86)
- Dense Structural Compression of Transformers via Gauge-Correct Channel Removal (NeurIPS 2026 · similarity 0.86)
- Peri-LN: Revisiting Normalization Layer in the Transformer Architecture (ICML 2025 · similarity 0.86)
- Transformer Layers as Painters (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.