Research map / Architectures and efficiency / Attention and transformers
Learning Correlation Structures for Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes structural self-attention (StructSA), which convolves query-key correlation maps to build dynamic aggregation kernels, and the StructViT architecture built on UniFormer that uses it, reaching state of the art on ImageNet-1K and several video benchmarks.
Problem
Vanilla self-attention uses individual query-key correlation values independently and ignores the structure of the correlation map, so it cannot capture spatial layouts or motion patterns in images and videos.
Approach
StructSA applies convolution over local query-key correlation maps to detect correlation patterns, then uses the resulting attention scores to dynamically aggregate local value contexts via D learned aggregation kernels, generalizing convolutional self-attention with convolutional projections.
Key results
- StructViT-B-4-1* reaches 86.5% top-1 on ImageNet-1K at 384 resolution with Token Labeling, versus 86.0 for UniFormer-B* at 384.
- StructViT-B-4-1 reaches 88.3% top-1 on Diving-48 and 54.2% per-class top-1 on FineGym Gym288, outperforming UniFormer-B* (87.4 and 53.5).
- On Something-Something V2, StructViT-B-4-1 reaches 71.5% top-1 versus 71.2% for UniFormer-B, with StructViT-S-8-1 gaining 1.3 points over UniFormer-S.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase structure dimension from D=0 to D=8 (ablation) | StructSA in DeiT-S · ImageNet-1K | top-1 accuracy: 80.5 -> 81.3 | Helped |
| Increase structure dimension from D=0 to D=8 (ablation) | StructSA in DeiT-S · Something-Something V1 | top-1 accuracy: 48.3 -> 50.6 | Helped |
| Use kernel size 1×1 instead of vanilla self-attention (ablation) | StructSA in DeiT-S · ImageNet-1K | top-1 accuracy: 80.5 -> 80.6 | No clear effect |
| Use kernel size 1×1 instead of vanilla self-attention (ablation) | StructSA in DeiT-S · Something-Something V1 | top-1 accuracy: 48.3 -> 48.5 | No clear effect |
| Use kernel size 3×3 instead of vanilla self-attention (ablation) | StructSA in DeiT-S · ImageNet-1K | top-1 accuracy: 80.5 -> 81.1 | Helped |
| Use kernel size 3×3 instead of vanilla self-attention (ablation) | StructSA in DeiT-S · Something-Something V1 | top-1 accuracy: 48.3 -> 50.4 | Helped |
| Use patch-wise rather than element-wise context aggregation (ablation) | StructSA in DeiT-S · ImageNet-1K | top-1 accuracy: 80.9 -> 81.1 | Helped |
| Use patch-wise rather than element-wise context aggregation (ablation) | StructSA in DeiT-S · Something-Something V1 | top-1 accuracy: 49.6 -> 50.4 | Helped |
| Replace UniFormer-S with StructViT-S-8-1 | StructViT · Kinetics-400 | top-1 accuracy: 80.8 -> 81.6 | Helped |
| Replace UniFormer-B with StructViT-B-4-1 | StructViT · Something-Something V1 | top-1 accuracy: 60.9 -> 61.3 | Helped |
| Replace UniFormer-B with StructViT-B-4-1 | StructViT · Something-Something V2 | top-1 accuracy: 71.2 -> 71.5 | Helped |
| Replace UniFormer-B* with StructViT-B-4-1 | StructViT · Diving-48 | top-1 accuracy: +0.9%p | Helped |
Limitations
- StructSA adds FLOPs over vanilla self-attention, requiring larger key/value strides to match baseline compute.
- Video models rely on ImageNet-1K and Kinetics-400 pretraining rather than larger pretraining corpora.
Experimental setup
Datasets: ImageNet-1K Kinetics-400 Something-Something V1 Something-Something V2 Diving-48 FineGym
Benchmarks: ImageNet-1K Kinetics-400 Something-Something V1 Something-Something V2 Diving-48 FineGym
Metrics: top-1 accuracy top-5 accuracy per-class top-1 accuracy parameters FLOPs
Baselines: UniFormer MViTv2 Swin ConvNext CvT MTV RSANet ORViT TimeSformer EfficientNet
Builds on: UniFormer vision transformer MViT Token Labeling conditional positional encoding
Closest papers by meaning
- Vision Transformers provably learn spatial structure (NeurIPS 2022 · similarity 0.87)
- Dissecting Query-Key Interaction in Vision Transformers (NeurIPS 2024 · similarity 0.85)
- Attention Guided CAM: Visual Explanations of Vision Transformer Guided by Self-Attention (AAAI 2024 · similarity 0.85)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.85)
- Unveiling the Hidden Structure of Self-Attention via Kernel Principal Component Analysis (NeurIPS 2024 · similarity 0.84)
- Towards Interpretable and Efficient Attention: Compressing All by Contracting a Few (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.