Research map / Label-efficient and robust learning / Self-supervised and contrastive
Efficient Self-supervised Vision Transformers for Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add region-level matching loss to the view-level objective (ablation) | EsViT Swin-T · ImageNet-1K validation | k-NN accuracy: 74.2 -> 75.7 | Helped |
| Add region-level matching loss to the view-level objective (ablation) | EsViT Swin-T/W=14 · ImageNet-1K validation | k-NN accuracy: 75.5 -> 77.0 | Helped |
| Add region-level matching loss to the view-level objective (ablation) | EsViT Swin-S · ImageNet-1K validation | k-NN accuracy: 76.8 -> 77.7 | Helped |
| Add region-level matching loss to the view-level objective (ablation) | EsViT Swin-S/W=14 · ImageNet-1K validation | k-NN accuracy: 77.3 -> 79.1 | Helped |
| Add region-level matching loss to the view-level objective (ablation) | EsViT Swin-B · ImageNet-1K validation | k-NN accuracy: 77.7 -> 78.9 | Helped |
| Increase window size from 7 to 14 (ablation) | EsViT Swin-T · ImageNet-1K validation | linear probe accuracy: 78.1 -> 78.7 | Helped |
| Increase window size from 7 to 14 (ablation) | EsViT Swin-S · ImageNet-1K validation | linear probe accuracy: 79.5 -> 80.8 | Helped |
| Use CvT sparse attention instead of Swin sparse attention (ablation) | EsViT Swin-T architecture comparison · ImageNet-1K validation, 300 epochs | linear probe accuracy: 75.3 -> 77.6 | Helped |
| Pre-train on ImageNet-22K rather than ImageNet-1K (ablation) | EsViT transfer representations · 18 Datasets | Scores: 80.66 -> 81.03 | Helped |
| Pre-train on a larger, less-curated dataset instead of ImageNet-1K (ablation) | EsViT transfer representations · ImageNet-1K | linear probe accuracy: 77.1 -> 75.4; 77.1 -> 69.6; 77.1 -> 73.5 | Hurt |
| Use the EsViT pre-training method rather than supervised pre-training | Swin-S · 18 small classification datasets | number of datasets outperforming supervised counterpart: 17 out of 18 | Helped |
| Use EsViT rather than supervised pre-training | Swin-T · COCO detection | AP^bb: 46.0 -> 46.2 | No clear effect |
Closest papers by meaning
- Revisiting Self-Supervised Visual Representation Learning (CVPR 2019 · similarity 0.91)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.91)
- A Closer Look at Self-Supervised Lightweight Vision Transformers (ICML 2023 · similarity 0.91)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.91)
- Patch-Level Representation Learning for Self-Supervised Vision Transformers (CVPR 2022 · similarity 0.89)
- You Only Need Less Attention at Each Stage in Vision Transformers (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.