Research map / Architectures and efficiency / RNNs and normalization
The Edge of Orthogonality: A Simple View of What Makes BYOL Tick
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace trainable linear predictor with closed-form NS^2 or Stiefel predictor (with ridge) | BYOL · ImageNet-1k linear, 100 epochs | top-1 accuracy: +1.2 (68.8 vs 67.6) | Helped |
| Replace trainable linear predictor with closed-form NS^2 or Stiefel predictor (with ridge) | BYOL · ImageNet-1k linear, 300 epochs | top-1 accuracy: +0.7 (72.2 vs 71.5) | Helped |
| Add ridge term alpha*I to the closed-form predictor (ablation) | DirectCopy predictor · ImageNet-1k linear, 100 epochs | top-1 accuracy: 20.9 at alpha=0 to ~67.5 at alpha=0.6 | Helped |
| Add ridge term alpha*I to the NE predictor (ablation) | NE predictor · ImageNet-1k linear, 100 epochs | top-1 accuracy: 61.2 at alpha=0 to 68.5 at alpha=0.15 | Helped |
| Add NE correction term (2 Z^T Z' - Z^T Z Z^T Z') relative to DirectCopy (ablation) | DirectCopy-style predictor · ImageNet-1k linear, 100 epochs, ridge 0.0 | top-1 accuracy: 61.2 vs 20.9 | Helped |
| Chain two Newton-Schulz iterations (NS^2) instead of one NS (9 iterations) (ablation) | NS predictor · ImageNet-1k linear, 100 epochs | top-1 accuracy: 68.8 vs 68.0 | Helped |
| Increase NS iterations from 3 to 9 (ridge 0.9) (ablation) | NS predictor · ImageNet-1k linear, 100 epochs | top-1 accuracy: 68.0 to 68.1 range | No clear effect |
| Reduce Visser iterations from 50+ to 12 or 25 (ablation) | Visser predictor · ImageNet-1k linear, 100 epochs | top-1 accuracy: 67.2 (12 iters) and 68.2 (25) vs 68.4 (50) | Hurt |
| Reduce Stiefel iterations from 9 to 3 (ablation) | Stiefel predictor · ImageNet-1k linear, 100 epochs | top-1 accuracy: 66.8 (3 iters) vs 68.8 (9 iters) | Hurt |
| Decrease EMA parameter tau toward 0 (remove EMA) (ablation) | BYOL · ImageNet-1k, 100 epochs | distance of predictor to polar factor | Hurt |
| Add ridge term to Stiefel predictor, stable across alpha>0 at 300 epochs (ablation) | Stiefel predictor · ImageNet-1k linear, 300 epochs | top-1 accuracy: 71.4 at alpha=0 vs 72.0-72.1 for alpha>0 | No clear effect |
Closest papers by meaning
- The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning (NeurIPS 2022 · similarity 0.86)
- PEIRA: Learning Predictive Encoders through Inter-View Regressor Alignment (NeurIPS 2026 · similarity 0.85)
- Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularization (NeurIPS 2024 · similarity 0.84)
- One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attention (ICLR 2024 · similarity 0.83)
- Cross-Entropy Is All You Need To Invert the Data Generating Process (ICLR 2025 · similarity 0.83)
- Deep linear networks for regression are implicitly regularized towards flat minima (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.