Research map / Architectures and efficiency / Attention and transformers
Evolving Attention with Residual Convolutions
Yujing Wang, Yaming Yang, Jiangang Bai, Mingliang Zhang, Jing Bai, Jing Yu, Ce Zhang, Gao Huang, Yunhai Tong
This paper tried a common change: Add residual connections
Add skip connections between attention maps · ImageNet · Top-1 accuracy: 74.29 -> 75.23 Improved
What 546 accepted papers report for this change
- 449 improved
- 35 no clear effect
- 11 mixed
- 51 hurt
- Resolution as a Direction: Vector-Panning Feature Alignment for Cross-Resolution Re-Identification ICML 2026 · add residual connection to VP · MLR-Market-1501 · Rank-1 (%): 91.2 -> 94.1Improved
- MAGIC: Multi-Granularity Language-Informed Image Clustering ICML 2026 · add residual connection to semantic adapters · CIFAR-20 · ACC: 66.1 -> 59.9Hurt
- Domain Adaptive Object Detection via Dynamic Causal Refinement ICML 2026 · add residual connection to SCC · Cityscapes to Foggy Cityscapes (0.02 split) · mAP@50: 64.0 -> 64.6Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add evolving attention to AA-ResNet | AA-ResNet · ImageNet | Top-1 accuracy: 74.33 -> 75.23; 77.70 -> 78.22; 78.50 -> 79.29; 79.10 -> 79.63 | Improved |
| Add evolving attention | BERT-Base · GLUE | Avg: 80.9 -> 83.3 | Improved |
| Add evolving attention | T5-Base · GLUE | Avg: 83.4 -> 84.5 | Improved |
| Add evolving attention | BERT-Large · GLUE | Avg: 83.7 -> 85.3 | Improved |
| Add evolving attention | RoBERTa-Large · GLUE | Avg: 86.4 -> 87.2 | Improved |
| Add evolving attention | Transformer-Lite and Transformer-Base · Machine translation | BLUE: 33.32 -> 33.80; 21.11 -> 21.63; 33.22 -> 34.12; 27.47 -> 27.56; 40.79 -> 41.54 | Improved |
| Add attention convolution (ablation) | EA-AA-ResNet-34 · ImageNet | Top-1 accuracy: 74.34 -> 75.23 | Improved |
| Add skip connections between attention maps (ablation) | EA-AA-ResNet-34 · ImageNet | Top-1 accuracy: 74.29 -> 75.23 | Improved |
| Use 3 × 3 rather than 1 × 1 convolution (ablation) | EA-AA-ResNet-34 · ImageNet | Top-1 accuracy: 74.99 -> 75.23 | Improved |
| Use 3 × 3 rather than 5 × 5 convolution (ablation) | EA-AA-ResNet-34 · ImageNet | Top-1 accuracy: 75.12 -> 75.23 | Mixed |
| Add attention convolution and skip connections (ablation) | EA-BERT-Base · CoLA | score: 52.1 -> 60.0; 52.9 -> 60.0 | Improved |
| Use 3 × 3 rather than 1 × 1 convolution (ablation) | EA-BERT-Base · CoLA | score: 57.5 -> 60.0 | Improved |
Closest papers by meaning
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.88)
- An Evolved Universal Transformer Memory (ICLR 2025 · similarity 0.88)
- Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape (ICML 2024 · similarity 0.88)
- LASER: Attention with Exponential Transformation (ICML 2025 · similarity 0.87)
- A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.87)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
