Research map / Architectures and efficiency / Attention and transformers
Delving Deep Into the Generalization of Vision Transformers Under Distribution Shifts
Chongzhi Zhang, Mingyuan Zhang, Shanghang Zhang, Daisheng Jin, Qiang Zhou, Zhongang Cai, Haiyu Zhao, Xianglong Liu, Ziwei Liu
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add prototype-based self-supervision and mutual information maximization | DeiT-B/16 · DomainNet | Avg. accuracy: 46.07 -> 51.07 | Improved |
| Add domain-adversarial training | DeiT-S/16 · DomainNet | Avg. accuracy: 43.22 -> 46.13 | Improved |
| Add minimax entropy training | DeiT-S/16 · DomainNet | Avg. accuracy: 43.22 -> 47.00 | Improved |
| Remove mutual information maximization from T-SSL (ablation) | DeiT-S/16 T-SSL · DomainNet | Avg. accuracy: 47.37 -> 46.35 | Hurt |
| Remove in-domain self-supervision from T-SSL (ablation) | DeiT-S/16 T-SSL · DomainNet | Avg. accuracy: 47.37 -> 47.30 | No clear effect |
| Use the complete T-SSL combination rather than either loss alone (ablation) | DeiT-S/16 · DomainNet | Avg. accuracy: 47.30, 46.35 -> 47.37 | Improved |
Closest papers by meaning
- The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancy (CVPR 2022 · similarity 0.88)
- Vision Transformer Neural Architecture Search for Out-of-Distribution Generalization: Benchmark and Insights (NeurIPS 2024 · similarity 0.88)
- Benchmarking Out-of-Distribution Generalization Capabilities of DNN-based Encoding Models for the Ventral Visual Cortex. (NeurIPS 2024 · similarity 0.87)
- Any-Shift Prompting for Generalization over Distributions (CVPR 2024 · similarity 0.86)
- Learning Where to Edit Vision Transformers (NeurIPS 2024 · similarity 0.86)
- Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
