Research map / Architectures and efficiency / Attention and transformers
A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a shallow ViT with trainable self-attention rather than a CNN of approximately the same model size | Binary classification under the paper's token data model · Binary classification under the paper's token data model | sample complexity: \Omega(\alpha_*^{-4}) for CNN; \Omega(\alpha_*^{-2}) for ViT | Helped |
| Have a sufficiently high fraction of label-relevant tokens relative to confusion tokens | Three-layer shallow ViT · Binary classification under the paper's token data model | generalization error: zero generalization error; \alpha_*\geq \alpha_# /c | Helped |
| Increase the fraction of label-relevant tokens, \alpha_* | Three-layer shallow ViT · Binary classification under the paper's token data model | sample complexity: N\geq \Omega(\alpha_*^{-2}) | Helped |
| Remove label-irrelevant tokens or tokens with large noise | Three-layer shallow ViT · Token sparsification | testing loss | Helped |
| Train the attention map for more iterations | Three-layer shallow ViT · Binary classification under the paper's token data model | attention weight mass on label-relevant tokens: 1-\eta^C; O(1/t) | Helped |
Closest papers by meaning
- Teaching Matters: Investigating the Role of Supervision in Vision Transformers (CVPR 2023 · similarity 0.90)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.90)
- Less Is More: Pay Less Attention in Vision Transformers (AAAI 2022 · similarity 0.90)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.90)
- Sparsifiner: Learning Sparse Instance-Dependent Attention for Efficient Vision Transformers (CVPR 2023 · similarity 0.90)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.