Research map / Label-efficient and robust learning / Self-supervised and contrastive
[Re] Understanding Self-Supervised Learning Dynamics without Contrastive Pairs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace gradient-trained linear predictor with DirectPred | BYOL linear predictor · ImageNet 300 epochs | top-1 accuracy: 69.9 -> 72.4 | Helped |
| replace gradient-trained linear predictor with DirectPred | BYOL linear predictor · ImageNet 60 epochs | top-1 accuracy: 59.4 -> 64.4 | Helped |
| DirectPred vs default 2-layer predictor | BYOL · ImageNet 300 epochs | top-1 accuracy: 72.5 -> 72.4 | No clear effect |
| replace gradient-trained regular predictor with DirectPred | BYOL linear predictor · STL-10 100 epochs | top-1 accuracy: 74.51 -> 76.77 | Helped |
| add regularization eps=0.1 to DirectPred (ablation) | DirectPred · STL-10 100 epochs | top-1 accuracy: 76.77 -> 77.86 | Helped |
| run eigen-decomposition every 5 minibatches with SGD in between (freq=5) (ablation) | DirectPred · STL-10 500 epochs | top-1 accuracy: 78.86 -> 80.28 | Helped |
| remove EMA with symmetric predictor (ablation) | BYOL symmetric W_p · STL-10 100 epochs | top-1 accuracy: 75.09 -> 36.62 | Hurt |
| remove EMA with regular two-layer predictor (ablation) | BYOL two-layer predictor · STL-10 100 epochs | top-1 accuracy: 78.85 -> 65.98 | Hurt |
| weight decay only on predictor restores symmetric W_p without EMA (ablation) | BYOL symmetric W_p, no EMA · STL-10 100 epochs | top-1 accuracy: 36.62 -> 71.12 | Helped |
| remove weight decay on all layers (ablation) | BYOL symmetric W_p, no EMA · STL-10 100 epochs | top-1 accuracy: 71.12 -> 43.04 | Hurt |
| plug in closed-form optimal predictor every minibatch instead of gradient training (ablation) | BYOL linear predictor, EMA · STL-10 100 epochs | top-1 accuracy: 74.51 -> 40.67 | Hurt |
| positive constant c_j=0.1 subtracted from eigenvalues with freq=1 (ablation) | DirectPred · STL-10 100 epochs | top-1 accuracy: 77.11 -> 46.57 | Hurt |
Closest papers by meaning
- Understanding self-supervised learning dynamics without contrastive pairs (ICML 2021 · similarity 0.89)
- On the Stepwise Nature of Self-Supervised Learning (ICML 2023 · similarity 0.87)
- Self-supervised contrastive learning performs non-linear system identification (ICLR 2025 · similarity 0.87)
- Data-Efficient Contrastive Self-supervised Learning: Most Beneficial Examples for Supervised Learning Contribute the Least (ICML 2023 · similarity 0.87)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.86)
- Collapse-Proof Non-Contrastive Self-Supervised Learning (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.