Research map / Label-efficient and robust learning / Self-supervised and contrastive
Harnessing small projectors and multiple views for efficient vision pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reformulates the augmentation-kernel SSL loss (shown equivalent to BarlowTwins/VICReg) to recommend low-dimensional projectors with stronger orthogonalization and more augmentations per image, yielding faster convergence and data efficiency on CIFAR, STL and ImageNet-100.
Problem
Existing non-contrastive SSL methods rely on heuristics (high-dimensional projectors, two augmentations per image) and are data- and compute-hungry, with little theory to guide practice.
Approach
Reformulates the augmentation-kernel spectral SSL objective via an operator T_M so it needs only O(m) rather than O(m^2) operations for m augmentations; analyzes gradient-descent implicit bias to justify low-dimensional projectors with beta proportional to 1/d, and uses multiple augmentations to better estimate the kernel.
Key results
- On CIFAR-10 with BarlowTwins, a 256-dim projector with tuned beta reaches 83.4% linear accuracy, matching or exceeding the 8192-dim projector (82.2%).
- Using 4 augmentations with a 256-dim projector reduces BarlowTwins time to reach baseline accuracy on CIFAR-10 from 99.36 to 43.09 min.
- With 4 augmentations on 50% of the data, VICReg reaches 80% CIFAR-10 accuracy in 32.39 min versus 66.05 min for the 2-augmentation 8192-dim baseline on full data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use low-dimensional projector (256 or 1024) with orthogonality coefficient tuned per dimension (beta ~ 1/d) instead of 8192-dim projector | BarlowTwins and VICReg · CIFAR-10 | linear probe accuracy (%): BT 83.4 (pdim 256, tuned beta) vs 82.2 (pdim 8192, fixed beta) | Helped |
| Reduce projector dimension to 64 while keeping the fixed beta from 8192-dim setting (ablation) | BarlowTwins and VICReg · CIFAR-10 | linear probe accuracy (%): BT 73.6 vs 82.2; VICReg 68.9 vs 80.4 | Hurt |
| Tune beta per projector dimension (64 pdim) rather than fixed beta (ablation) | BarlowTwins · CIFAR-10 | linear probe accuracy (%): 73.6 -> 82.1 | Helped |
| Use 4 augmentations instead of 2 with 256-dim projector (ablation) | BarlowTwins and VICReg · CIFAR-10 | time to reach 80% accuracy (min): BT 62.34 -> 43.09; VICReg 51.73 -> 39.02 (2 augs pdim 256 vs 4 augs) | Helped |
| Use 4 augmentations with 50% of unique data (same effective dataset size) instead of 2 augs full data (ablation) | VICReg · CIFAR-10 | time to 80% accuracy (min): VICReg 66.05 (2 augs pdim 8192, 100%) -> 32.39 (4 augs, 50%) | Helped |
| Increase augmentations to 8 with 25% of unique data (ablation) | BarlowTwins · CIFAR-10 | time to 80% accuracy (min): 63.43 (2 augs, 100%, pdim 8192) -> 27.74 (8 augs, 25%) | Helped |
| Use more augmentations (8) with very small data fraction (25%) for VICReg (ablation) | VICReg · CIFAR-10 | time to 80% accuracy (min): VICReg 8 augs 25% = 34.76 vs 4 augs 50% = 32.39 | Mixed |
Surprising findings
- Low-dimensional projectors (256-dim) with a stronger orthogonality coefficient match the performance of 8192-dim projectors, with optimal beta scaling roughly as 1/d.
- More augmentations on a fraction of the dataset can match or beat two augmentations on the full dataset, with gains lost in the scarce-data regime.
Limitations
- Multiple augmentations add per-step compute overhead, so wall-clock gains are smaller than epoch-level gains.
- Fewer unique samples with many augmentations fails to improve performance in the scarce-data regime.
- Theory relies on small initialization and linear/NTK approximations; applied only to BarlowTwins/VICReg-style losses.
Experimental setup
Datasets: CIFAR-10 STL-10 ImageNet-100
Benchmarks: CIFAR-10 STL-10 ImageNet-100
Metrics: linear probe accuracy time to reach 80% accuracy (min)
Baselines: Barlow Twins VICReg SimCLR SWaV
Builds on: Barlow Twins VICReg HaoChen et al. augmentation graph spectral contrastive loss Garrido et al. contrastive-non-contrastive duality Simon et al. eigenlearning dynamics
Closest papers by meaning
- Beyond Random Augmentations: Pretraining with Hard Views (ICLR 2025 · similarity 0.90)
- Views Can Be Deceiving: Improved SSL Through Feature Space Augmentation (ICLR 2024 · similarity 0.89)
- Can Generative Models Improve Self-Supervised Representation Learning? (AAAI 2025 · similarity 0.88)
- Improving Self-Supervised Learning by Characterizing Idealized Representations (NeurIPS 2022 · similarity 0.88)
- Self-supervised Representation Learning from Random Data Projectors (ICLR 2024 · similarity 0.87)
- Probing the Mid-level Vision Capabilities of Self-Supervised Learning (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.