Research map / Architectures and efficiency / Attention and transformers
On the Integration of Self-Attention and Convolution
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace PVT-T, PVT-S, Swin-T, or Swin-S attention blocks with ACmix | Transformer backbones with segmentation heads · ADE20K | val mIoU: 37.1 -> 42.7; 42.4 -> 46.4; 44.5 -> 45.3; 47.6 -> 48.7 | Helped |
| Use ACmix in the compared detection backbones | ResNet 50, SAN 19, PVT-T, and Swin-T · COCO detection | mAP: 36.7 -> 38.3; 38.2 -> 39.1; 36.7 -> 40.5; 46.0 -> 47.0; 50.5 -> 51.1 | Helped |
| Use ACmix instead of PVT-S or PVT-L | PVT backbones · COCO RetinaNet inference on Ascend 910 | FPS: 50.3 -> 67.7; 24.3 -> 43.9 | Helped |
| Use ACmix instead of PVT-S | PVT-S backbone · COCO RetinaNet inference on Ascend 910 | mAP: 42.2 -> 42.0 | Hurt |
| Combine self-attention and convolution paths with independently learned weights (ablation) | Swin-ACmix-T · ImageNet-1K | top-1: 81.3 -> 81.9 | Helped |
| Fix both path weights to 1 instead of learning them independently (ablation) | Swin-ACmix-T · ImageNet-1K | top-1: 81.9 -> 81.5 | Hurt |
| Constrain learned path weights to sum to 1 (ablation) | Swin-ACmix-T · ImageNet-1K | top-1: 81.9 -> 81.5 | Hurt |
| Replace tensor shifts with fixed-kernel group convolution (ablation) | ACmix convolution path · Swin-Transformer-T, RTX2080Ti | FPS: 313 -> 419 | Helped |
| Replace fixed shift kernels with learnable kernels initialized randomly (ablation) | ACmix convolution path · ImageNet-1K, Swin-Transformer-T | Top-1: 81.4 -> 81.5 | No clear effect |
| Use learnable group-convolution kernels with designed shift initialization (ablation) | ACmix convolution path · ImageNet-1K, Swin-Transformer-T | Top-1: 81.4 -> 81.9 | Helped |
Closest papers by meaning
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.88)
- KernelWarehouse: Rethinking the Design of Dynamic Convolution (ICML 2024 · similarity 0.88)
- Revisiting the Integration of Convolution and Attention for Vision Backbone (NeurIPS 2024 · similarity 0.88)
- Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis (NeurIPS 2024 · similarity 0.87)
- AsCAN: Asymmetric Convolution-Attention Networks for Efficient Recognition and Generation (NeurIPS 2024 · similarity 0.87)
- Faster Neighborhood Attention: Reducing the O(n^2) Cost of Self Attention at the Threadblock Level (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.