Research map / Architectures and efficiency / Attention and transformers
DeepCrossAttention: Supercharging Transformer Residual Connections
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeepCrossAttention (DCA), which replaces fixed residual sums in transformers with learnable input-dependent weighted combinations of prior layer outputs plus depth-wise cross-attention for queries/keys/values, and reports lower perplexity and up to 3x faster training on LM1B/C4 with a theoretical low-rank analysis.
Problem
Traditional residual connections simply sum previous layer outputs, which dilutes information from individual layers and creates a bottleneck in transformer training and capacity.
Approach
Generalize residual connections (GRN-v1/v2/v3) to learnable, dimension- and input-dependent weighted combinations of all prior layer outputs, and in each transformer block use three independent GRNs to produce the attention queries, keys, and values from the layer-output stack (DCA).
Key results
- On LM1B, DCA with a 6-layer model reaches 18.06 perplexity vs 18.98 for the transformer and 18.65 for Hyper-Connections, the best prior method.
- The 2-DCA 24-layer model reaches the transformer's final perplexity (15.14) in about one third of the training time (normalized time 0.33).
- On ViT-S/16 ImageNet classification, ViT+DCA improves accuracy from 76.4 to 77.1 (+0.7%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DCA to the Transformer | 6-layer Transformer · LM1B | perplexity: 18.98±0.01 -> 18.06±0.01 | Helped |
| Add DCA to the Transformer | 13-layer Transformer, embedding dimension 1111 · C4 | perplexity: 21.534 -> 20.392 | Helped |
| Add DCA to ViT-S/16 | ViT-S/16 · ImageNet | accuracy: +0.7% (76.4 -> 77.1) | Helped |
| Restrict the GRN stack to first and last-1 layer outputs (ablation) | DCA · LM1B | perplexity: 15.14±0.06 -> 14.48±0.05 | Helped |
| Restrict the GRN stack to first and last-2 layer outputs (ablation) | DCA · LM1B | perplexity: 15.14±0.06 -> 14.41±0.04 | Helped |
| Restrict the GRN stack to first and last-4 layer outputs (ablation) | DCA · LM1B | perplexity: 15.14±0.06 -> 14.50±0.03 | Helped |
| Restrict the GRN stack to first and last-8 layer outputs (ablation) | DCA · LM1B | perplexity: 15.14±0.06 -> 14.49±0.02 | Helped |
| Restrict the GRN stack to first and last-16 layer outputs (ablation) | DCA · LM1B | perplexity: 15.14±0.06 -> 14.35±0.07 | Helped |
| Restrict the GRN stack to first and last-24 layer outputs (ablation) | DCA · LM1B | perplexity: 15.14±0.06 -> 14.35±0.00 | Helped |
| Use three independent GRN-v3 instances per decoder block (DCA) rather than GRN-v1 (ablation) | GRN-v1 · LM1B | perplexity: 18.80±0.11 -> 18.06±0.01 | Helped |
| Use three independent GRN-v3 instances per decoder block (DCA) rather than GRN-v2 (ablation) | GRN-v2 · LM1B | perplexity: 18.43±0.04 -> 18.06±0.01 | Helped |
| Use three independent GRN-v3 instances per decoder block (DCA) rather than GRN-v3 (ablation) | GRN-v3 · LM1B | perplexity: 18.41±0.10 -> 18.06±0.01 | Helped |
Surprising findings
- Adding DCA to an already pre-trained transformer for 60k extra steps improves perplexity by 0.17, while continuing the plain transformer gives only 0.02.
- The benefit of DCA shrinks as model width grows (perplexity delta from -2.82 at width 64 to -0.39 at width 1024), while the depth-scaling benefit persists.
Limitations
- Gains diminish with larger width; the theory is for a stylized low-rank linear model and only extends to nonlinear models via bottleneck-rank arguments.
- Efficiency: the full layer-output stack grows with depth, so the k-DCA approximation is needed to keep cost manageable.
Experimental setup
Datasets: LM1B C4 ImageNet
Benchmarks: LM1B C4 ImageNet
Metrics: perplexity accuracy loss training speed normalized training time
Baselines: Transformer LAuReL-PA DenseFormer (1x1) Hyper-Connections ViT
Builds on: DenseFormer ResNet DenseNet Hyper-Connections
Closest papers by meaning
- Transformer Layers as Painters (AAAI 2025 · similarity 0.85)
- MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections (ICML 2025 · similarity 0.85)
- Residual Stream Analysis with Multi-Layer SAEs (ICLR 2025 · similarity 0.85)
- Viewing Transformers Through the Lens of Long Convolutions Layers (ICML 2024 · similarity 0.84)
- SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention (NeurIPS 2024 · similarity 0.84)
- Dynamic Layer Tying for Parameter-Efficient Transformers (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.