Research map / Label-efficient and robust learning / Self-supervised and contrastive
Understanding the Learning Phases in Self-Supervised Learning via Critical Periods
JangHyeon Lee, Philipe Ambrozio Dias, Yao-Yi Chiang, Dalton Lunga
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select CP closure checkpoint instead of final checkpoint | VICReg-RN50-fMoW · fMoW-WILDS (OOD) | top-1 accuracy: 0.341 -> 0.430 | Improved |
| select CP closure checkpoint instead of final checkpoint | VICReg-RN50-fMoW · fMoW-val (ID) | top-1 accuracy: 0.621 -> 0.610 | Hurt |
| select CP closure checkpoint instead of final checkpoint | VICReg-RN50-fMoW · EuroSAT-Spatial (OOD) | top-1 accuracy: 0.894 -> 0.912 | Improved |
| add CP-guided self-distillation of early layers to final checkpoint | VICReg-RN50-fMoW · fMoW-WILDS (OOD) | top-1 accuracy: 0.341 -> 0.445 | Improved |
| add CP-guided self-distillation of early layers to final checkpoint | VICReg-RN50-fMoW · EuroSAT (OOD) | top-1 accuracy: 0.917 -> 0.944 | Improved |
| add CP-guided self-distillation of early layers to final checkpoint | VICReg-RN50-fMoW · fMoW-val (ID) | top-1 accuracy: 0.621 -> 0.617 | No clear effect |
| distill early layers rather than all layers (ablation) | CP-SD · fMoW-WILDS (OOD) | top-1 accuracy: 0.421 -> 0.445 | Improved |
| distill early layers rather than all layers (ablation) | CP-SD · EuroSAT-Spatial (OOD) | top-1 accuracy: 0.908 -> 0.925 | Improved |
| inject noise deficit early in pretraining rather than late | SSL pretraining (SimCLR, VICReg, DINO, MAE on fMoW) · fMoW-val | normalized sensitivity | Hurt |
| increase minibatch reuse from k=1 to k=5 (ablation) | DINO pretraining · Fisher Information | FI peak timing | Mixed |
| use ViT-Base instead of ViT-Small (ablation) | DINO pretraining · Fisher Information | FI stabilization | Mixed |
| pretrain longer (to 1000 epochs) instead of stopping at CP closure | SSL pretraining · fMoW-WILDS (OOD) | top-1 accuracy: 0.430 -> 0.341 | Hurt |
Closest papers by meaning
- Exploring Transferability of Self-Supervised Learning by Task Conflict Calibration (AAAI 2026 · similarity 0.88)
- What shapes the loss landscape of self supervised learning? (ICLR 2023 · similarity 0.87)
- In-Context Learning Strategies Emerge Rationally (NeurIPS 2025 · similarity 0.86)
- Memorization in Self-Supervised Learning Improves Downstream Generalization (ICLR 2024 · similarity 0.86)
- Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning (ICLR 2025 · similarity 0.86)
- Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
