Research map / Label-efficient and robust learning / Self-supervised and contrastive
On the Effectiveness of Supervision in Asymmetric Non-Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends asymmetric non-contrastive learning (SimSiam/BYOL) to supervised settings (SupSiam/SupBYOL) by adding a label-based target pool loss, showing supervision reduces intra-class variance and that balancing supervised and self-supervised losses via alpha is key.
Problem
Supervision has been well studied for contrastive learning, but its extension to asymmetric non-contrastive learning (ANCL) for supervised representation learning is underexplored, and its theoretical effect is unclear.
Approach
Add a supervised loss that pulls the predictor output of an anchor toward the averaged stop-gradient targets of same-label samples drawn from a target pool (queue or class-wise queues/prototypes), combined convexly with the self-supervised ANCL loss; a linear-model analysis shows this reduces intra-class variance by a factor alpha.
Key results
- SupBYOL achieves 87.43% top-1 linear probing on ImageNet-100 vs 84.93% for BYOL and 87.40% for SupCon; SupSiam reaches 87.31% vs 82.15% for SimSiam.
- On 11 transfer linear-evaluation datasets, SupBYOL attains the best average rank (1.36) and SupSiam ranks second (2.27), outperforming SupCon (4.73).
- On the toy dataset and ImageNet-100, alpha between 0 and 1 (e.g., 0.5) gives the best transfer performance, while alpha=1 (no supervision) is worst.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add supervised targets to SimSiam | SimSiam · ImageNet-100 | Top-1 accuracy: 82.15 -> 87.31 | Helped |
| Add supervised targets to BYOL | BYOL · ImageNet-100 | Top-1 accuracy: 84.93 -> 87.43 | Helped |
| Add supervised targets to SimSiam | SimSiam · VOC07 test | AP: 53.56 { $\pm$ 0.10} -> 53.89 { $\pm$ 0.26} | Helped |
| Add supervised targets to SimSiam | SimSiam · VOC07 test | AP50: 79.82 { $\pm$ 0.10} -> 80.28 { $\pm$ 0.06} | Helped |
| Add supervised targets to ANCL methods | SimSiam and BYOL · ImageNet-100 downstream transfer (11 datasets) | Average rank: 4.91 -> 2.27; 3.82 -> 1.36 | Helped |
| Set α to 0.5 rather than α=1.0 (ablation) | SupSiam · Toy dataset | Linear probing accuracy: 45.91 -> 61.89 | Helped |
| Set α to 0.5 rather than α=1.0 (ablation) | SupSiam · ImageNet-100 | Linear probing accuracy: 82.15 -> 87.31 | Helped |
| Set α to 0.5 rather than α=1.0 (ablation) | SupSiam · CUB200 | Mean per-class accuracy: 36.10 -> 43.48 | Helped |
| Set α to 0.5 rather than α=1.0 (ablation) | SupSiam · Dogs | Mean per-class accuracy: 54.57 -> 64.65 | Helped |
| Use class-wise queues with 20 features per class instead of a class-agnostic pool (ablation) | SupSiam · ImageNet-100 downstream transfer | Average accuracy: 70.40 -> 70.44 | No clear effect |
| Remove the momentum network while retaining the target pool (ablation) | Supervised ANCL · CIFAR100 | k-NN: 74.55 -> 73.92 | Hurt |
| Remove the target pool while retaining the momentum network (ablation) | Supervised ANCL · CIFAR100 | k-NN: 74.55 -> 73.32 | Hurt |
Surprising findings
- Excessively reducing intra-class variance (alpha near 0) hurts generalization and fine-grained transfer, so pure supervision is not optimal.
- Supervised ANCL gains over self-supervised ANCL are relatively small on generic transfer, but large on fine-grained datasets such as CUB200, Dogs and Pets.
Limitations
- Theoretical analysis relies on linear encoder/predictor and Gaussian class-conditional assumptions.
- Experiments are mostly on ImageNet-100 with 200-epoch pretraining and batch size 128; scale to full ImageNet is not shown.
- Target pool design (class-wise queues or learnable prototypes) requires extra memory and hyperparameters such as alpha and pool size.
Experimental setup
Datasets: ImageNet-100 CIFAR100 toy Gaussian dataset
Benchmarks: ImageNet-100 linear probing VOC object detection CIFAR10 CIFAR100 DTD Food MIT67 SUN397 Caltech101 CUB200 Stanford Dogs Flowers
Metrics: top-1 accuracy linear probing accuracy AP AP50 few-shot accuracy k-NN accuracy
Baselines: SimCLR SupCon MoCo-v2 SupMoCo SimSiam BYOL
Builds on: SimSiam BYOL SupCon MoCo-v2
Closest papers by meaning
- Bridging the Gap from Asymmetry Tricks to Decorrelation Principles in Non-contrastive Self-supervised Learning (NeurIPS 2022 · similarity 0.86)
- On the Alignment Between Supervised and Self-Supervised Contrastive Learning (ICLR 2026 · similarity 0.85)
- On the duality between contrastive and non-contrastive self-supervised learning (ICLR 2023 · similarity 0.85)
- Does Learning from Decentralized Non-IID Unlabeled Data Benefit from Self Supervision? (ICLR 2023 · similarity 0.85)
- Understanding self-supervised learning dynamics without contrastive pairs (ICML 2021 · similarity 0.84)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.