Research map / Architectures and efficiency / Attention and transformers
Unraveling Attention via Convex Duality: Analysis and Interpretations of Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a linear MLP-Mixer head instead of the strongest linear convex baseline | CIFAR-100 transfer learning with fixed gMLP-S backbone · CIFAR-100 | Top-1 accuracy: 65.95 -> 78.11 | Helped |
| Use a ReLU MLP-Mixer head instead of the strongest ReLU convex baseline | CIFAR-100 transfer learning with fixed gMLP-S backbone · CIFAR-100 | Top-1 accuracy: 73.05 -> 80.22 | Helped |
| Replace a linear B-FNO head with a ReLU B-FNO head (ablation) | B-FNO · CIFAR-100 | Top-1 accuracy: 68.68 -> 77.65 | Helped |
| Replace a linear self-attention head with a ReLU self-attention head (ablation) | Self-Attention · CIFAR-100 | Top-1 accuracy: 73.81 -> 74.74 | No clear effect |
| Replace a linear MLP-Mixer head with a ReLU MLP-Mixer head (ablation) | MLP-Mixer · CIFAR-100 | Top-1 accuracy: 78.11 -> 80.22 | No clear effect |
| Replace a linear FNO head with a ReLU FNO head (ablation) | FNO · CIFAR-100 | Top-1 accuracy: 72.29 -> 72.93 | No clear effect |
| Use a ReLU MLP head instead of a linear MLP head (ablation) | Standard convex MLP · CIFAR-100 | Top-1 accuracy: 65.95 -> 73.05 | Helped |
Closest papers by meaning
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.88)
- On the Role of Attention Masks and LayerNorm in Transformers (NeurIPS 2024 · similarity 0.88)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.88)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.87)
- Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers (Student Abstract) (AAAI 2024 · similarity 0.87)
- Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.