Research map / Label-efficient and robust learning / Self-supervised and contrastive
An Investigation of Representation and Allocation Harms in Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that underrepresenting a class makes contrastive learning representations collapse toward semantically similar majority groups (representation harm), that this mediates downstream allocation harm, and gives a stochastic block model theory of the collapse.
Problem
Underrepresentation of groups in self-supervised contrastive learning data may collapse minority-group representations onto similar majority groups, causing representation harm and downstream allocation harm; this had been underexplored.
Approach
Defines a representation harm metric (ratio of average cosine distances between groups under underrepresented vs balanced CL training), links it to allocation harm via causal mediation analysis (natural indirect effect), and proves neural-collapse-style asymptotics of the CL loss under a stochastic block model with group imbalance.
Key results
- On CIFAR10 with automobile underrepresented to 1%, the representation harm between automobile and truck is 0.78 ± 0.009, indicating collapse toward the semantically similar class.
- Mediation analysis shows automobiles are mistaken as trucks most often when underrepresented (TE 0.798 ± 0.002), with the highest NIE at 0.062 ± 0.001 attributable to representation harm.
- On BiasBios, GRH(attorney, paralegal) is 0.94, meaning female attorneys' representations are closer to female paralegals than to male attorneys.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Undersample automobile to 1% during SimCLR training | SimCLR on CIFAR10 · CIFAR10 | representation harm (RH), automobile-truck: 0.78 ± 0.009 | Hurt |
| Undersample either automobile or truck to 1% | SimCLR on CIFAR10 · CIFAR10 | representation harm (RH), automobile-truck: 0.78 ± 0.009; 0.832 ± 0.012 | Hurt |
| Undersample a class during SimCLR training | SimCLR on CIFAR10 · CIFAR10 | natural indirect effect (NIE): 0.062 ± 0.001 | Hurt |
| Use SimSiam instead of SimCLR | Contrastive learning on CIFAR10 · CIFAR10 | representation harm (RH): 0.834 ± 0.025; 0.882 ± 0.0 | Hurt |
| Use boosted contrastive learning instead of SimCLR | SimCLR on CIFAR10 · CIFAR10 | representation harm (RH), automobile-truck: 0.78 ± 0.009 and 0.832 ± 0.012 -> 0.829 ± 0.001 and 0.897 ± 0.001 | Helped |
| Use boosted contrastive learning instead of SimCLR | SimCLR on CIFAR10 · CIFAR10 | diagonal NIE: 5.3± 0.2 %; 3.7 ± 0.01% | Hurt |
| Undersample the first block in the connected-block stochastic block model | Contrastive learning on stochastic block model · stochastic block model | cosine similarity between connected groups: 0.52 (± 0.08) -> 0.7 (± 0.07); 0.83 (± 0.03) | Hurt |
| Reduce the first block's connectivity with the second block | Contrastive learning on stochastic block model · stochastic block model | cosine similarity between group representations: 0.4; 0.1 | Helped |
| Undersample the minority gender within occupations during SimCSE training | SimCSE on BiasBios · BiasBios | gender representation harm (GRH): 0.94; 0.95 | Hurt |
| Balance gender representation when training the logistic-regression predictor | SimCSE representations with logistic regression on BiasBios · BiasBios | gender allocation harm (GAH) | Helped |
| Balance class frequencies in addition to gender when training the logistic-regression predictor | SimCSE representations with logistic regression on BiasBios · BiasBios | gender allocation harm (GAH) | No clear effect |
| Balance gender and class frequencies when training the logistic-regression predictor | SimCSE representations with logistic regression on BiasBios · BiasBios | allocation harm for underrepresented occupations | Helped |
Surprising findings
- Boosted contrastive learning does not fully eliminate representation harm: automobile-truck RH only rises to 0.829 and 0.897, and accuracy drops of 5.3% and 3.7% remain.
- Reweighing for gender parity in the downstream classifier does not fully repair allocation harms caused by CL-stage underrepresentation.
Limitations
- Theoretical analysis uses a layer-peeled assumption and an asymptotic SBM setting, not finite-sample neural networks
- Empirical studies are limited to CIFAR10 with a single subsampled class setting and one natural text dataset
- Mediation analysis is infeasible for BiasBios because a balanced reference model cannot be trained
Experimental setup
Datasets: CIFAR10 BiasBios
Benchmarks: CIFAR10 BiasBios
Metrics: representation harm (RH) ratio of cosine distances gender representation harm (GRH) gender allocation harm (GAH) total effect (TE) natural indirect effect (NIE) classification accuracy
Baselines: SimCLR balanced model SimSiam boosted contrastive learning (BCL_I)
Builds on: SimCLR SimSiam SimCSE node2vec boosted contrastive learning
Closest papers by meaning
- A theoretical study of inductive biases in contrastive learning (ICLR 2023 · similarity 0.89)
- Self-Damaging Contrastive Learning (ICML 2021 · similarity 0.88)
- Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppression (ICML 2023 · similarity 0.88)
- How does Labeling Error Impact Contrastive Learning? A Perspective from Data Dimensionality Reduction (ICML 2025 · similarity 0.88)
- Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learning (NeurIPS 2025 · similarity 0.86)
- Separating common from salient patterns with Contrastive Representation Learning (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.