Research map / Label-efficient and robust learning / Self-supervised and contrastive
Views Can Be Deceiving: Improved SSL Through Feature Space Augmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that SSL augmentations create spurious connectivity that biases downstream linear probes, and proposes LateTVG, which prunes late encoder layers to create feature-space views that improve worst-group accuracy without group labels.
Problem
Self-supervised pretraining on unlabeled data with spurious correlations can learn representations that rely on spurious features, hurting worst-group accuracy of downstream linear classifiers; standard augmentations and group-based resampling do not fix this.
Approach
Theoretical toy analysis shows augmentations can create spurious connectivity in the augmentation graph; LateTVG generates a second view by passing the input through a magnitude-pruned copy of the encoder (late layers pruned), inducing invariance to late-layer features without requiring group or label information.
Key results
- On MetaShift, SSL-LateTVG raises SimSiam worst-group accuracy from 42.3% to 79.6% (SSL-base vs. LateTVG).
- On Spurious CIFAR-10, worst-group accuracy improves from 43.4% to 61.4% with SimSiam; on Waterbirds from 48.3% to 56.3%.
- Resampling during SSL pretraining does not consistently help: e.g., balancing gives -8.3 points worst-group on Spurious CIFAR-10 and +3.0 on Waterbirds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add pruning-based feature-space view generation during SimSiam pretraining | SimSiam representations · CelebA | worst-group accuracy: 77.5 -> 83.1 | Helped |
| Add pruning-based feature-space view generation during SimSiam pretraining | SimSiam representations · CMNIST | worst-group accuracy: 80.7 -> 83.1 | Helped |
| Add pruning-based feature-space view generation during SimSiam pretraining | SimSiam representations · MetaShift | worst-group accuracy: 42.3 -> 79.6 | Helped |
| Add pruning-based feature-space view generation during SimSiam pretraining | SimSiam representations · Spurious CIFAR-10 | worst-group accuracy: 43.4 -> 61.4 | Helped |
| Add pruning-based feature-space view generation during SimSiam pretraining | SimSiam representations · Waterbirds | worst-group accuracy: 48.3 -> 56.3 | Helped |
| Balance groups during SSL pretraining | SimSiam · CelebA, CMNIST, MetaShift, Spurious CIFAR-10, Waterbirds | worst-group accuracy difference: -1.7; -8.7; -3.8; -8.3; +3.0 | Mixed |
| Downsample majority groups during SSL pretraining | SimSiam · CelebA, CMNIST, MetaShift, Spurious CIFAR-10, Waterbirds | worst-group accuracy difference: +0.3; -10.6; +3.9; -14.4; +0.5 | Mixed |
| Upsample minority groups during SSL pretraining | SimSiam · CelebA, CMNIST, MetaShift, Spurious CIFAR-10, Waterbirds | worst-group accuracy difference: +4.1; -5.3; +2.7; -19.4; -0.3 | Mixed |
| Use L1 instead of no downstream linear-probe regularization (ablation) | SimSiam features · CelebA | worst-group accuracy: 66.1 -> 77.5 | Helped |
| Use L2 instead of no downstream linear-probe regularization (ablation) | SimSiam features · MetaShift | worst-group accuracy: 41.8 -> 45.6 | Helped |
| Increase pruning rate / vary pruning threshold (ablation) | SSL-LateTVG on Hard ImageNet · Hard ImageNet Gray split | accuracy: 59.5; 59.1; 56.1; 50.7 | Hurt |
| Increase pruning rate / vary pruning threshold (ablation) | SSL-LateTVG on Hard ImageNet · Hard ImageNet Gray BBox split | accuracy: 51.1; 49.6; 48.0; 42.4 | Hurt |
Surprising findings
- Resampling groups during SSL pretraining does not consistently improve downstream worst-group accuracy and can drop it substantially (e.g., -19.4 points for upsampling on Spurious CIFAR-10).
- Even with group-balanced resampling, SSL representations still show high spurious connectivity, implying augmentations rather than data imbalance drive the bias.
Limitations
- Evaluated only on small spurious-correlation benchmarks, not large-scale SSL corpora like ImageNet.
- Primarily tested with SimSiam; expected to work best with Siamese encoders with stop-gradient.
- Theoretical analysis is limited to a simplified toy augmentation graph with assumptions.
Experimental setup
Datasets: CelebA Waterbirds CMNIST Spurious CIFAR-10 MetaShift Hard ImageNet
Benchmarks: CelebA Waterbirds CMNIST Spurious CIFAR-10 MetaShift Hard ImageNet
Metrics: worst-group accuracy average accuracy spurious connectivity error
Baselines: SSL-Base (SimSiam) SSL-Base (SimCLR) Supervised pretraining (ERM) Balancing/Downsampling/Upsampling resampling
Builds on: SimSiam SimCLR spectral contrastive learning Just Train Twice Deep Feature Reweighting
Closest papers by meaning
- Harnessing small projectors and multiple views for efficient vision pretraining (NeurIPS 2024 · similarity 0.89)
- Can Generative Models Improve Self-Supervised Representation Learning? (AAAI 2025 · similarity 0.88)
- Measuring Self-Supervised Representation Quality for Downstream Classification Using Discriminative Features (AAAI 2024 · similarity 0.87)
- Joint‑Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self‑Supervised Learning (NeurIPS 2025 · similarity 0.86)
- Self-supervised Representation Learning from Random Data Projectors (ICLR 2024 · similarity 0.86)
- Adv-SSL: Adversarial Self-Supervised Representation Learning with Theoretical Guarantees (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.