Research map / Architectures and efficiency / Attention and transformers
Linear Differential Vision Transformer: Learning Visual Contrasts via Pairwise Differentials
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Visual-Contrast Attention (VCA), a linear-complexity drop-in replacement for MHSA that pools queries into positive/negative visual-contrast tokens and applies differential attention, improving DeiT, hierarchical ViTs, and DiT/SiT generators.
Problem
Standard multi-head self-attention in Vision Transformers costs O(N^2 C) and spends computation on weak or redundant token correlations; differential attention from language is also quadratic and ignores image structure.
Approach
Average-pool each head's query field into a small set of visual-contrast tokens, split them into learnable positive and negative streams via distinct positional embeddings, and compute a two-stage differential attention (global contrast map, then patch-wise query attention) so cost scales as O(N n C) with n << N.
Key results
- DeiT-Tiny top-1 on ImageNet-1K rises from 72.2% to 75.6% (+3.4) with VCA and no extra FLOPs.
- VCA improves PVT-T by up to 3.1 points and Swin/CSwin by 0.4 to 1.0 points.
- On class-conditional ImageNet 256x256 generation, VCA lowers FID-50K by 2.1 to 5.2 points across DiT and SiT Small and Base models with patch sizes 8, 4, and 2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace MHSA with VCA | DeiT-T · ImageNet-1K val | Top-1 accuracy: 72.2 -> 75.6 | Helped |
| Replace attention blocks with VCA | DiT-S/2 · ImageNet-1K 256×256 | FID-50K: 67.2 -> 62.3 | Helped |
| Replace attention blocks with VCA | SiT-S/2 · ImageNet-1K 256×256 | FID-50K: 57.3 -> 53.0 | Helped |
| Use VCA in Stage I only (ablation) | DeiT-Tiny attention · ImageNet-1K | Top-1 accuracy: 75.1 -> 75.5 | Helped |
| Use VCA in Stage II only (ablation) | DeiT-Tiny attention · ImageNet-1K | Top-1 accuracy: 75.1 -> 75.4 | Helped |
| Use VCA in both stages instead of differential attention (ablation) | DeiT-Tiny attention · ImageNet-1K | Top-1 accuracy: 75.1 -> 75.6 | Helped |
| Use VCA in Stage I only (ablation) | DiT-S/2 attention · ImageNet-1K 256×256 | FID-50K: 63.9 -> 64.3 | Helped |
| Use VCA in Stage II only (ablation) | DiT-S/2 attention · ImageNet-1K 256×256 | FID-50K: 63.9 -> 64.6 | Hurt |
| Use VCA in both stages instead of differential attention (ablation) | DiT-S/2 attention · ImageNet-1K 256×256 | FID-50K: 63.9 -> 62.3 | Helped |
| Use pooled query features with separate positional embeddings in both streams instead of learned embeddings (ablation) | DeiT-Tiny visual-contrast tokens · ImageNet-1K | Top-1 accuracy: 75.1 -> 75.6 | Helped |
| Use pooled query features with separate positional embeddings in both streams instead of learned embeddings (ablation) | DiT-S/2 visual-contrast tokens · ImageNet-1K 256×256 | FID-50K: 63.7 -> 62.3 | Helped |
| Replace the positive stream's pooled features plus positional embedding with pooled features only (ablation) | DeiT-Tiny visual-contrast tokens · ImageNet-1K | Top-1 accuracy: 75.6 -> 75.5 | Hurt |
Surprising findings
- Replacing only Stage I or only Stage II with VCA already captures most of the gain, with the two branches combining nearly additively.
- Using pooled queries for the positive stream instead of learnable embeddings gives a larger FID gain than the reverse substitution, suggesting real image statistics matter more in the positive branch.
Limitations
- Task-agnostic average pooling may miss edge-rich or fine details.
- The added micro-attention may reduce speed gains on small images.
- Extensions to video, 3D, and language tasks are unexplored.
Experimental setup
Datasets: ImageNet-1K
Benchmarks: ImageNet-1K
Metrics: top-1 accuracy FID-50K parameters FLOPs
Baselines: DeiT PVT Swin CSwin DiT SiT Differential Transformer (Diff)
Builds on: Differential Transformer Agent Attention DeiT DiT SiT
Closest papers by meaning
- Bridging the Divide: Reconsidering Softmax and Linear Attention (NeurIPS 2024 · similarity 0.88)
- Breaking the Low-Rank Dilemma of Linear Attention (CVPR 2025 · similarity 0.88)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.88)
- LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate (NeurIPS 2024 · similarity 0.87)
- LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision (NeurIPS 2025 · similarity 0.87)
- CARE Transformer: Mobile-Friendly Linear Visual Transformer via Decoupled Dual Interaction (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.