Research map / Label-efficient and robust learning / Self-supervised and contrastive
Teaching Matters: Investigating the Role of Supervision in Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use dense feature extraction instead of normal feature extraction (ablation) | ViT models on DAVIS · DAVIS | J and F Mean: 0.18 -> 0.39; 0.34 -> 0.58; 0.17 -> 0.38; 0.34 -> 0.59; 0.51 -> 0.68; 0.34 -> 0.56; 0.19 -> 0.41; 0.35 -> 0.60; 0.38 -> | Helped |
| Use FS rather than other compared supervision methods | ViT-B/16 · ImageNet k-NN classification and ROxford5k (M) | Top-1 accuracy and mAP: 83.79 (12); 0.45 (12) | Helped |
| Use MoCo rather than other compared supervision methods | ViT-B/16 · DAVIS | J and F Mean: 0.61 (11) | Helped |
| Use CLIP rather than other compared supervision methods | ViT-B/16 · SPair-71k | PCK@0.1: 30.70 (8) | Helped |
| Use contrastive self-supervision rather than reconstruction-based self-supervision | ViTs · PartImageNet part-level feature clustering | Feature purity | Helped |
| Use contrastive self-supervision rather than explicit supervision | ViTs · DAVIS video segmentation | J and F Mean: 0.61 vs 0.59; 0.60 vs 0.59 | Helped |
| Use reconstruction-based self-supervision rather than explicit supervision | ViTs · DAVIS video segmentation | J and F Mean: 0.58 vs 0.59; 0.54 vs 0.59 | Mixed |
| Use the final layer rather than selecting a task-appropriate intermediate layer | ViTs · Localized downstream tasks | Task performance | Hurt |
| Change supervision method | ViTs · Attention local/global processing across layers | Average Attention Distance | Mixed |
| Use explicit supervision | ViTs · Late-layer attention maps | Attention alignment with salient content | Hurt |
| Use reconstruction-based pretraining | MAE and BEiT ViTs · Image-level CLS feature clustering | Cluster purity | Hurt |
| Use contrastive self-supervision rather than reconstruction-based self-supervision | ViTs · Object-level feature clustering on COCO | Feature purity | Helped |
Closest papers by meaning
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.91)
- Attention Transfer Is Not Universally Effective for Vision Transformers (NeurIPS 2026 · similarity 0.91)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.91)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.91)
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.90)
- Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.