Research map / Label-efficient and robust learning / Self-supervised and contrastive
On the Out-of-Distribution Generalization of Self-Supervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a PID-based mini-batch sampling strategy for SSL, using a regularized latent variable model and propensity-score matching, to reduce spurious correlations and improve OOD generalization.
Problem
SSL models learn spurious correlations within mini-batch tasks, which degrades out-of-distribution generalization.
Approach
Defines a post-intervention distribution (PID) where the spurious latent variable and the anchor label are independent, shows SSL trained on PID-satisfying batches has minimax-optimal worst-case risk, and builds a mini-batch sampler that approximates PID via a learned regularized latent variable model and propensity-score matching.
Key results
- On ImageNet-100 linear evaluation, Barlow Twins + Ours reaches 77.77% top-1 vs 75.97% for Barlow Twins, and VICRegL + Ours reaches 78.20% top-1.
- On ImageNet 1% semi-supervised (1000 epochs), BYOL + Ours reaches 58.9% top-1 vs 56.3% for BYOL.
- On miniImageNet (5,1) few-shot, SwAV + Ours reaches 49.15% vs 44.39% for SwAV.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace random mini-batch sampling with PID sampling | VICRegL · ImageNet-100 | Top-1 accuracy: 75.96 -> 78.20 | Helped |
| replace random mini-batch sampling with PID sampling | VICRegL · ImageNet, 400 epochs | Top-1 accuracy: 72.14 -> 74.91 | Helped |
| replace random mini-batch sampling with PID sampling | BYOL · ImageNet 1% labels, 1000 epochs | Top-1 accuracy: 56.3 -> 58.9 | Helped |
| replace random mini-batch sampling with PID sampling | MoCo · ImageNet 10% labels, 200 epochs | Top-1 accuracy: 61.9 -> 71.4 | Helped |
| replace random mini-batch sampling with PID sampling | SimSiam · VOC 07 detection | AP: 48.5 -> 50.8 | Helped |
| replace random mini-batch sampling with PID sampling | SwAV · miniImageNet (5,5) | accuracy: 54.91 -> 64.28 | Helped |
| vary the original batch size (ablation) | BYOL and BYOL + Ours · ImageNet | performance | Mixed |
| vary sampling hyperparameter a (ablation) | PID sampling strategy · ImageNet with BYOL | performance | No clear effect |
| vary orthogonality weight α across [0.001,0.01,0.1,1,10] (ablation) | PID sampling strategy · ImageNet-100 | performance | Mixed |
Surprising findings
- Some baselines show decreased accuracy with the method on certain settings, e.g., MoCo + Ours top-1 on ImageNet 400 epochs is below the base in some rows is not observed, but SimSiam + Ours on ImageNet 1000 epochs (73.67) is below SwAV + Ours; the method's gains are not uniform across settings.
Limitations
- Identifiability of the spurious variable s is not theoretically proven; relies on an assumption.
- Sampling cost grows quadratically in dataset size D for matching, and the method is sensitive to the sub-batch size a.
- Theoretical guarantees rely on strong assumptions (e.g., exact balancing score matching and environment diversity).
Experimental setup
Datasets: ImageNet ImageNet-100 CIFAR-10 CIFAR-100 STL-10 Tiny ImageNet PASCAL VOC COCO Omniglot miniImageNet CIFAR-FS
Benchmarks: ImageNet-100 ImageNet ImageNet 1% and 10% semi-supervised PASCAL VOC detection COCO detection and instance segmentation Omniglot few-shot miniImageNet few-shot CIFAR-FS few-shot
Metrics: top-1 accuracy top-5 accuracy AP AP50 AP75 few-shot accuracy
Baselines: SimCLR MoCo SimSiam Barlow Twins SwAV DINO RELIC v2 VICRegL BYOL
Builds on: SimCLR MAE VAE balancing score / propensity score Prototype Networks
Closest papers by meaning
- Out-of-Distribution Generalization With Causal Invariant Transformations (CVPR 2022 · similarity 0.88)
- Towards a Theoretical Framework of Out-of-Distribution Generalization (NeurIPS 2021 · similarity 0.87)
- Out-of-Variable Generalisation for Discriminative Models (ICLR 2024 · similarity 0.86)
- Strengthen Out-of-Distribution Detection Capability with Progressive Self-Knowledge Distillation (ICML 2025 · similarity 0.86)
- Causality Inspired Federated Learning for OOD Generalization (ICML 2025 · similarity 0.86)
- Aggregation Hides Out-of-Distribution Generalization Failures from Spurious Correlations (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.