Research map / Label-efficient and robust learning / Self-supervised and contrastive
Understanding self-supervised learning dynamics without contrastive pairs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace gradient-trained linear prediction with DirectPred | BYOL on ImageNet · ImageNet, 300 epochs | Top-1 accuracy: 69.9 -> 72.4 | Helped |
| Replace the BYOL two-layer predictor with DirectPred | BYOL on ImageNet · ImageNet, 300 epochs | Top-1 accuracy: 72.5 -> 72.4 | No clear effect |
| Replace gradient-trained linear prediction with DirectPred | BYOL on ImageNet · ImageNet, 60 epochs | Top-1 accuracy: 59.4% vs. 64.4% | Helped |
| Use DirectPred with gradient updates between eigen-decompositions (freq=5) (ablation) | DirectPred on STL-10 · STL-10, 500 epochs | Top-1 accuracy: 78.86±1.15 -> 80.28±0.62 | Helped |
| Set epsilon regularization to 0.1 rather than 0 (ablation) | DirectPred on STL-10 · STL-10, Top-1, 100 epochs; rho=0.3 | accuracy: 76.77±0.24 -> 77.86±0.16 | Helped |
| Set epsilon to 0.5 (ablation) | DirectPred on STL-10 · STL-10, Top-1, 100 epochs; rho=0.3 | accuracy: 77.86±0.16 -> 75.06±1.10 | Hurt |
| Remove EMA while constraining the predictor to be symmetric (ablation) | BYOL on STL-10 · STL-10, Top-1, 100 epochs | accuracy: 75.09±0.48 -> 36.62±1.85 | Hurt |
| Remove EMA while retaining a regular, asymmetric predictor (ablation) | BYOL on STL-10 · STL-10, Top-1, 100 epochs | accuracy: 74.51±0.47 -> 72.85±0.16 | Hurt |
| Remove trunk weight decay while keeping predictor weight decay (ablation) | BYOL with symmetric predictor and no EMA · STL-10, Top-1, 100 epochs | accuracy: 43.04±2.32 -> 71.12 ±0.71 | Helped |
| Remove weight decay from all layers (ablation) | BYOL with symmetric predictor and no EMA · STL-10, Top-1, 100 epochs | accuracy: 71.12 ±0.71 -> 43.04±2.32 | Hurt |
| Remove the stop-gradient operation (ablation) | SimSiam · Linear-model analysis | representational collapse | Hurt |
| Remove the predictor (ablation) | BYOL and SimSiam · Non-contrastive SSL | representational collapse | Hurt |
Closest papers by meaning
- On the duality between contrastive and non-contrastive self-supervised learning (ICLR 2023 · similarity 0.91)
- Improving Self-Supervised Learning by Characterizing Idealized Representations (NeurIPS 2022 · similarity 0.90)
- PEIRA: Learning Predictive Encoders through Inter-View Regressor Alignment (NeurIPS 2026 · similarity 0.89)
- Self-supervised contrastive learning performs non-linear system identification (ICLR 2025 · similarity 0.89)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.88)
- Collapse-Proof Non-Contrastive Self-Supervised Learning (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.