Research map / Architectures and efficiency / Attention and transformers
Transformer Meets Twicing: Harnessing Unattended Residual Information
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Twicing Attention, which replaces softmax self-attention with (2A - A^2)V via a kernel twicing procedure, slowing representation collapse and improving accuracy and robustness on ImageNet, ADE20K, and WikiText-103.
Problem
Transformer self-attention representations degrade (representation collapse / over-smoothing) across layers, reducing token diversity and performance.
Approach
Interpret self-attention as a low-pass NLM smoothing filter (Nadaraya-Watson kernel) and replace the attention operator A with the twicing operator 2A - A^2, which slows eigenvalue decay and reuses smoothed residual information; implemented as A V + A(V - A V) at O(N^2 D) cost.
Key results
- DeiT-Twicing reaches 72.60 top-1 on ImageNet vs 72.00 for DeiT, and under PGD attack 9.15 vs 8.16 top-1.
- On WikiText-103 the Transformer-Twicing small model reduces test PPL from 37.51 to 36.69 and attacked PPL from 55.17 to 54.46.
- FAN-Twicing improves FAN on ImageNet top-1 (77.18 vs 77.09) and ImageNet-C mCE (60.48 vs 60.71), with ADE20K mIoU rising from 34.73 to 35.12 for DeiT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard self-attention with Twicing Attention | DeiT · ImageNet-1K | Top-1 accuracy: 72.00 -> 72.60 | Helped |
| Replace standard self-attention with Twicing Attention | DeiT · ImageNet PGD, FGSM, and SPSA | Top-1 accuracy: 8.16 -> 9.15; 29.88 -> 32.28; 66.41 -> 67.12 | Helped |
| Replace standard self-attention with Twicing Attention | DeiT · ImageNet-R, ImageNet-A, ImageNet-C, ImageNet-C (Extra) | Robustness metric (Top-1 accuracy or mCE): 32.22 -> 32.74; 6.97 -> 7.66; 72.21 -> 70.33; 63.68 -> 62.46 | Mixed |
| Replace standard self-attention with Twicing Attention | FAN · ImageNet-1K clean and adversarial tests | Top-1 accuracy: 77.09 -> 77.18; 11.91 -> 12.80; 33.81 -> 35.52; 67.15 -> 68.89 | Helped |
| Replace standard self-attention with Twicing Attention | DeiT segmentation model · ADE20K | Mean IoU: 34.73 -> 35.12 | Helped |
| Replace standard self-attention with Twicing Attention | Transformer (small) · WikiText-103 test | Perplexity: 37.51 -> 36.69 | Helped |
| Replace standard self-attention with Twicing Attention | Transformer · WikiText-103 attacked test | Perplexity: 55.17 -> 54.46 | Helped |
| Use Twicing Attention during fine-tuning | Switch Transformer · SST-2 | Test accuracy: 77.78 -> 78.34 | Helped |
| Apply Twicing Attention in layers 1-12 rather than standard attention throughout (ablation) | DeiT · ImageNet-1K | Top-1 accuracy: 72.00 -> 72.60 | Helped |
| Apply Twicing Attention in layers 7-12 rather than standard attention throughout (ablation) | DeiT · ImageNet-1K | Top-1 accuracy: 72.00 -> 72.45 | Helped |
| Apply Twicing Attention only in layers 10-12 rather than standard attention throughout (ablation) | DeiT · ImageNet-1K | Top-1 accuracy: 72.00 -> 72.31 | Helped |
Surprising findings
- Applying Twicing Attention only in the last three layers gives most of the gains with negligible extra compute, and early-layer-only placement offers little improvement.
- Twicing Attention adds no parameters, yet the paper reports performance gains consistently across tasks.
Limitations
- Applying Twicing Attention to all layers incurs some extra computation relative to baseline.
- Theoretical analysis relies on approximations (Monte-Carlo, Stirling asymptotics) and the eigencapacity proxy.
- Gains on some robustness metrics (e.g., ImageNet-A, FAN-Twicing SPSA-free categories) are mixed or marginal.
Experimental setup
Datasets: ImageNet-1K ADE20K WikiText-103 SST-2 ImageNet-A ImageNet-R ImageNet-C
Benchmarks: ImageNet-1K ImageNet-A ImageNet-R ImageNet-C ADE20K WikiText-103 SST-2
Metrics: top-1 accuracy top-5 accuracy mCE pixel accuracy mean IoU perplexity cosine similarity between tokens GFLOPs ms/it
Baselines: DeiT NeuTRENO FAN FeatScale Transformer (WikiText-103 baseline) Linear Transformer
Builds on: Transformer DeiT Fully Attentional Network nonlocal means denoising twicing kernels
Closest papers by meaning
- Attention-Only Transformers via Unrolled Subspace Denoising (ICML 2025 · similarity 0.87)
- Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.87)
- Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Models (ICML 2024 · similarity 0.86)
- AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformers (ICML 2024 · similarity 0.86)
- Understanding Differential Transformer Unchains Pretrained Self-Attentions (NeurIPS 2025 · similarity 0.86)
- Theory, Analysis, and Best Practices for Sigmoid Self-Attention (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.