Research map / Label-efficient and robust learning / Recognition and long-tail
Mitigating Label Bias in Machine Learning: Fairness through Confident Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a model-agnostic data selection method that uses truncated (robust mean) confidence thresholds from confident learning plus co-teaching-style group-split models to filter label-biased instances and train fairer classifiers.
Problem
Label bias in training data, where unbiased labels are overwritten by biased annotators, causes classifiers to inherit discrimination; standard confident learning wrongly flags low-confidence, underrepresented-group instances as label errors.
Approach
Train two networks on demographic-group subsets, estimate per-class confidence thresholds using a truncated influence-function (robust M-estimator) mean with confidence-interval correction, and use the resulting confident joint to remove off-diagonal instances before retraining a classifier on the remaining data.
Key results
- On Adult under 20% symmetric bias, the method reaches 15.75% test error vs 18.89% for label correction (LC) and 22.61% for confident learning, with DEO violation 0.12.
- On Law under 20% symmetric bias, the method reaches 9.09% error and 0.05 DEO violation, versus 9.54% error for LC.
- Truncated threshold (T) reduces error at 40% symmetric synthetic bias to 0.91% versus 3.60% for the mean estimator (M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace mean-based confidence thresholding with truncated thresholding (ablation) | Confident Learning data selection on synthetic data · Synthetic, symmetric label bias | test error: 1.51 -> 0.77; 3.60 -> 0.91 | Helped |
| replace mean-based confidence thresholding with truncated thresholding (ablation) | Confident Learning data selection on synthetic data · Synthetic, asymmetric label bias | test error: 0.45 -> 0.36; 0.55 -> 0.38 | Helped |
| replace mean-based confidence thresholding with truncated thresholding (ablation) | Confident Learning data selection on synthetic data · Synthetic, symmetric and asymmetric label bias | fairness violation: 0.01 -> 0.00 | Helped |
| increase label-bias magnitude from 20% to 40% | Ours · Real-world datasets, symmetric bias | test error | Mixed |
Surprising findings
- Plain confident learning does not consistently yield fairer classifiers than label bias correction methods, attributed to class-dependent noise assumptions that ignore demographics.
- LongReMix achieves the lowest DEO on COMPAS but has high test error because it predicts mostly negative outcomes; its p% and DP violations are high.
Limitations
- Performance is not uniformly best; on some settings (e.g., COMPAS, Credit) baselines such as LC or LongReMix match or beat it on some fairness metrics.
- Fairness gains are sometimes accompanied by higher DP or lower p% in 40% bias settings, as the authors acknowledge.
- Evaluation limited to a two-layer MLP and small-scale tabular datasets.
Experimental setup
Datasets: Synthetic Adult ProPublica COMPAS Credit Loan Data Law School Admissions
Benchmarks: Adult COMPAS Law School Admissions Credit Loan Data
Metrics: test error (%) difference of equal opportunity (DEO) demographic parity distance p%
Baselines: confident learning (CL) LongReMix label bias correction (LC) group peer loss (GPL)
Builds on: confident learning co-teaching LongReMix group peer loss
Closest papers by meaning
- On Group Sufficiency Under Label Bias (NeurIPS 2025 · similarity 0.90)
- Navigating Towards Fairness with Data Selection (AAAI 2025 · similarity 0.89)
- FairNet: Dynamic Fairness Correction without Performance Loss via Contrastive Conditional LoRA (NeurIPS 2025 · similarity 0.87)
- How Far Can Fairness Constraints Help Recover From Biased Data? (ICML 2024 · similarity 0.87)
- Subgroups Matter for Robust Bias Mitigation (ICML 2025 · similarity 0.86)
- Fairness without Harm: An Influence-Guided Active Sampling Approach (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.