Research map / Label-efficient and robust learning / Recognition and long-tail
DeNetDM: Debiasing by Network Depth Modulation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeNetDM, which trains deep (biased) and shallow (debiased) branches as a product of experts and then distills the shallow debiased knowledge into a target-architecture debiased model, without bias labels or augmentation.
Problem
Neural networks trained on biased data learn spurious correlations and generalize poorly; existing debiasing methods need bias labels or augmentation/reweighting of bias-conflicting samples.
Approach
Theoretically and empirically links network depth to the rank of encoded attributes (deeper nets favor lower-rank, bias attributes), then exploits depth modulation within a product-of-experts training scheme: a deep biased branch and a shallow debiased branch, followed by distillation into a target debiased model.
Key results
- On Colored MNIST at 0.5% bias-conflicting ratio, DeNetDM reaches 74.72% vs 63.48% for LC (best baseline).
- On BAR at 1% conflicting ratio, DeNetDM achieves 73.84% vs 70.94% for LC; on BFFHQ 75.7% vs 70.0% for LC.
- On CelebA worst-group accuracy, DeNetDM reports 81.04% vs 76.80% for JTT, without bias labels or augmentation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DeNetDM instead of the strongest listed baseline | CMNIST · CMNIST, 0.5–5.0% bias-conflicting | accuracy: 74.72 (0.99) vs 63.48 (5.22); 85.22 (0.76) vs 78.41 (1.95); 89.29 (0.51) vs 83.63 (1.43); 93.54 (0.22) vs 88.29 (1.50) | Helped |
| Use DeNetDM instead of the strongest listed baseline | C-CIFAR10 · C-CIFAR10, 0.5–5.0% bias-conflicting | accuracy: 38.93 (1.16) vs 34.56 (0.69); 44.20 (0.77) vs 38.30; 47.35 (0.70) vs 47.81 (2.00); 56.30 (0.42) vs 57.32 | Mixed |
| Use DeNetDM instead of the strongest listed baseline | BAR · BAR, 1.0% and 5.0% bias-conflicting | accuracy: 73.84 (2.56) vs 70.94 (1.46); 79.61 (3.18) vs 74.32 (2.42) | Helped |
| Use DeNetDM instead of the strongest listed baseline | BFFHQ · BFFHQ, 1.0% bias-conflicting | accuracy: 75.7 (2.8) vs 70.0 (1.4) | Helped |
| Use DeNetDM instead of the strongest listed baseline | CelebA · CelebA | worst-group accuracy: 81.04 vs 76.80 | Helped |
| Use DeNetDM instead of LC | CivilComments · CivilComments | worst-group accuracy: 68.33 (-) vs 70.3 (1.2) | Hurt |
| Add the Stage-2 target loss while omitting distillation (ablation) | DeNetDM · C-CIFAR10, 1% bias-conflicting | accuracy: 37.47 -> 42.89 | Helped |
| Add the Stage-2 distillation loss while omitting the target loss (ablation) | DeNetDM · C-CIFAR10, 1% bias-conflicting | accuracy: 37.47 -> 42.25 | Helped |
| Add the target loss to CE plus distillation (ablation) | DeNetDM · C-CIFAR10, 1% bias-conflicting | accuracy: 42.25 -> 43.12 | Helped |
| Add distillation to CE plus the target loss (ablation) | DeNetDM · C-CIFAR10, 1% bias-conflicting | conflicting accuracy: 35.74 -> 39.46 | Helped |
Surprising findings
- Setting both branches to the same depth makes the branches' bias/core separation unreliable, with large variance in accuracy across seeds.
- On CivilComments, DeNetDM (68.33%) is below LC (70.3%) and JTT (69.3%) in worst-group accuracy.
Limitations
- Relies on hyperparameter tuning using a small unbiased validation set with bias annotations.
- Theoretical results rely on simplifying assumptions (e.g., stable partitioning, two attributes, heuristic probability bounds).
Experimental setup
Datasets: Colored MNIST Corrupted CIFAR-10 Biased FFHQ BAR CelebA CivilComments
Benchmarks: Colored MNIST Corrupted CIFAR-10 BFFHQ BAR CelebA CivilComments
Metrics: test accuracy worst-group accuracy aligned accuracy conflicting accuracy
Baselines: ERM Group DRO LfF JTT DFA LC
Builds on: Product of Experts Learning from Failure (LfF) Just Train Twice (JTT) Learning Debiased Representation via Disentangled Feature Augmentation (DFA) Simplicity bias
Closest papers by meaning
- Self-supervised Debiasing Using Low Rank Regularization (CVPR 2024 · similarity 0.89)
- SEBRA : Debiasing through Self-Guided Bias Ranking (ICLR 2025 · similarity 0.86)
- Diffusing DeBias: Synthetic Bias Amplification for Model Debiasing (NeurIPS 2025 · similarity 0.86)
- A Simple Remedy for Dataset Bias via Self-Influence: A Mislabeled Sample Perspective (NeurIPS 2024 · similarity 0.85)
- Enhancing Intrinsic Features for Debiasing via Investigating Class-Discerning Common Attributes in Bias-Contrastive Pair (CVPR 2024 · similarity 0.85)
- Disentangling, Amplifying, and Debiasing: Learning Disentangled Representations for Fair Graph Neural Networks (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.