Research map / Architectures and efficiency / Attention and transformers
Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows via gradient descent analysis of a two-layer softmax-attention ViT on a signal-plus-noise token model that benign vs. harmful overfitting is separated by N·SNR² thresholds.
Problem
Understanding when Vision Transformers generalize well while overfitting (benign overfitting) versus when they memorize harmful noise, which prior theory left largely unexplored for softmax attention trained from random initialization.
Approach
Analyzes gradient descent on a two-layer ViT (softmax self-attention + fixed linear layer) with a patch-based signal/noise data model, using vectorized Q&K and scalarized V decompositions and a three-stage decoupling to characterize training dynamics and derive a sharp phase transition in N·SNR².
Key results
- Theory: benign overfitting (small test loss) when N·SNR² = Ω(1); harmful overfitting (test loss Θ(1)) when N^{-1}·SNR^{-2} = Ω(1).
- Synthetic experiments show the empirical boundary between test loss above and below 0.2 closely follows the theoretical curve N·SNR² = 1000.
- MNIST experiments show a transition between benign and harmful overfitting regimes as N and SNR vary.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the training sample size and signal-to-noise ratio | Two-layer Transformer · Synthetic data | test loss | Helped |
| Increase the training sample size and signal-to-noise ratio | Vision Transformer · MNIST | test loss | Helped |
| Train in the regime N · SNR² = Ω(1) rather than N⁻¹ · SNR⁻² = Ω(1) | Two-layer Transformer · Synthetic data | test loss | Helped |
Limitations
- Proof relies on sparsity of attention, requiring the noise scale tilde-sigma_p to be much larger than sigma_p.
- Simplified model: two-layer transformer with fixed linear output layer and M=Θ(1) tokens, far from full ViT.
- Strong over-parameterization and specific initialization/learning-rate conditions are required.
Experimental setup
Datasets: synthetic signal-plus-noise token data MNIST (with added Gaussian noise borders)
Benchmarks: synthetic data test loss heatmap MNIST test loss heatmap
Metrics: test loss training loss
Builds on: feature learning theory benign overfitting in CNNs (ReLU^q) Vision Transformer
Closest papers by meaning
- Unraveling the Gradient Descent Dynamics of Transformers (NeurIPS 2024 · similarity 0.88)
- Benign Overfitting in Single-Head Attention (NeurIPS 2025 · similarity 0.88)
- Improving Interpretation Faithfulness for Vision Transformers (ICML 2024 · similarity 0.87)
- Trained Transformer Classifiers Generalize and Exhibit Benign Overfitting In-Context (ICLR 2025 · similarity 0.87)
- A General and Efficient Training for Transformer via Token Expansion (CVPR 2024 · similarity 0.87)
- Linear attention is (maybe) all you need (to understand Transformer optimization) (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.