Research map / Label-efficient and robust learning / Self-supervised and contrastive
Self-supervised Debiasing Using Low Rank Regularization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeFund, a self-supervised debiasing framework where a biased encoder pretrained with a rank (feature-decorrelation) regularizer identifies bias-conflicting samples that are upweighted to debias the main model.
Problem
Debiasing deep networks against spurious correlations when annotations of bias attributes and target labels are scarce or absent, i.e., learning debiased models from unlabeled or minimally labeled data.
Approach
Spurious correlations lower the effective rank of latent representations; deliberately enforcing rank reduction via an auto-correlation off-diagonal penalty creates a biased encoder (semantic bottleneck) that is used to mine bias-conflicting samples, which are then upweighted when training the main model.
Key results
- On CelebA (makeup) linear evaluation, DeFund reaches 58.4% bias-conflict accuracy vs 46.9% for SimCLR, with unbiased accuracy 73.1% vs 69.8%.
- On UTKFace (age) linear evaluation, DeFund improves bias-conflict accuracy to 59.5% from SimCLR's 36.4%, outperforming ERM (45.4%) and LNL/EnD.
- In semi-supervised CelebA (makeup, 10% labels), DeFund achieves 70.6% bias-conflict accuracy and 78.8% unbiased accuracy, vs JTT at 55.6% and 75.7%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DeFund instead of SimCLR | self-supervised linear evaluation · UTKFace (age), UTKFace (gender), CelebA (makeup) | bias-conflict accuracy: 36.4 -> 59.5; 56.3 -> 63.7; 46.9 -> 58.4 | Helped |
| use DeFund instead of SimCLR | self-supervised linear evaluation · UTKFace (age), UTKFace (gender), CelebA (makeup) | unbiased accuracy: 66.3 -> 70.6; 74.2 -> 74.9; 69.8 -> 73.1 | Helped |
| use DeFund instead of SimCLR | semi-supervised learning with 10% labels · CelebA (makeup), CelebA (blonde) | bias-conflict accuracy: 50.5±4.7 -> 60.5±0.4; 70.6±1.0 -> 75.1±0.8 | Helped |
| add rank regularization to SimCLR (ablation) | self-supervised linear evaluation · UTKFace (age), UTKFace (gender), CelebA (makeup) | bias-conflict accuracy: 36.4 -> 26.6; 56.3 -> 50.9; 46.9 -> 43.9 | Hurt |
| upweight the main model's error set without a biased model (ablation) | SimCLR linear evaluation · UTKFace (age), UTKFace (gender), CelebA (makeup) | bias-conflict accuracy: 36.4 -> 53.0; 56.3 -> 58.3; 46.9 -> 50.1 | Helped |
| combine rank regularization with biased-model sample mining and upweighting (ablation) | SimCLR linear evaluation · UTKFace (age), UTKFace (gender), CelebA (makeup) | bias-conflict accuracy: 36.4 -> 59.5; 56.3 -> 63.7; 46.9 -> 58.4 | Helped |
Surprising findings
- Adding rank regularization to a model that is meant to be unbiased amplifies bias: rank-regularized models lose bias-conflict accuracy while keeping bias-aligned accuracy.
- Non-contrastive self-supervised methods (SimSiam, VICReg) perform worse than contrastive SimCLR as main encoders, though they may be viable for the biased model.
Limitations
- Overall performance may be upper-bounded by the constraint on effective dimensionality from rank regularization, sacrificing bias-conflict accuracy.
- Evaluated mainly on a limited set of biased datasets with binary tasks and ResNet-18 backbones.
- Hyperparameter (lambda_reg, lambda_up) selection relies on reporting the best bias-conflict accuracy among models with improved unbiased accuracy, which uses bias-conflict validation information.
Experimental setup
Datasets: UTKFace CelebA MIMIC-CXR NIH ChestX-ray Waterbirds Colored MNIST Corrupted CIFAR-10 MultiCMNIST
Benchmarks: UTKFace CelebA MIMIC-CXR + NIH Waterbirds MultiCMNIST
Metrics: bias-conflict accuracy unbiased accuracy precision recall
Baselines: ERM SimCLR VICReg SimSiam LNL EnD LfF JTT CVaR DRO DebiAN
Builds on: SimCLR JTT LfF effective rank Barlow Twins-style decorrelation
Closest papers by meaning
- DeNetDM: Debiasing by Network Depth Modulation (NeurIPS 2024 · similarity 0.89)
- Debiased Self-Training for Semi-Supervised Learning (NeurIPS 2022 · similarity 0.88)
- SEBRA : Debiasing through Self-Guided Bias Ranking (ICLR 2025 · similarity 0.87)
- A Simple Remedy for Dataset Bias via Self-Influence: A Mislabeled Sample Perspective (NeurIPS 2024 · similarity 0.87)
- Enhancing Intrinsic Features for Debiasing via Investigating Class-Discerning Common Attributes in Bias-Contrastive Pair (CVPR 2024 · similarity 0.86)
- Diffusing DeBias: Synthetic Bias Amplification for Model Debiasing (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.