Research map / Label-efficient and robust learning / Noisy labels
Regretful Decisions under Label Noise
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces instance-level regret and an "ambiguity" measure for learning under label noise, estimated by training models on plausible clean datasets sampled from the posterior noise model, and shows its use for data cleaning and selective prediction on clinical and enhancer datasets.
Problem
Standard noise-robust learning achieves good population-level performance but leaves individual predictions unreliable: each instance is subject to a "lottery of mistakes" due to label noise, and practitioners cannot tell where the model errs.
Approach
Defines regret for individual predictions under noisy labels (expected regret equals the posterior noise rate), shows hedging methods still incur regret, and estimates instance-level "ambiguity" by training models on plausible clean datasets drawn from typical noise draws under the posterior noise model, then uses it for data cleaning and abstention.
Key results
- Under 5% label noise, over half of instances are susceptible to regret across all five clinical datasets (susceptibility about 52-53%).
- Hedging reduces true error relative to ignoring noise (e.g., mortality LR at 40% noise), but regret stays unchanged at about 19.5% of instances.
- On enhancer selective classification, disagreement-based abstention improves hit rate by 10.7% over standard confidence-based abstention at 20% abstention.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Hedge against label noise instead of ignoring it | Logistic regression on lungcancer with 40% positive-label noise · lungcancer | True error: 43.0% -> 31.1% | Helped |
| Hedge against label noise instead of ignoring it | Logistic regression on lungcancer with 40% positive-label noise · lungcancer | Regret: 19.7% -> 19.7% | No clear effect |
| Drop high-ambiguity training instances before training the final model | Data cleaning on shock_mimic · shock_mimic, clean-label test | Test error: 14.9% reduction | Helped |
| Use ambiguity rather than standard confidence to decide abstention | Selective classification on cshock_mimic · cshock_mimic, 20% label noise; 40% abstention | Selective error: -6.6% | Helped |
| Use ambiguity rather than standard confidence to decide abstention | Selective classification on cshock_mimic · cshock_mimic, 20% label noise; 40% abstention | Selective regret: -5.9% | Helped |
| Use disagreement among plausible models rather than predicted-probability confidence for abstention | Enhancer experiment selection · enhancer, 20% abstention | Hit rate: +10.7% | Helped |
Surprising findings
- Hedging lowers clean error yet leaves regret unchanged, redistributing mistakes rather than removing them.
- Even 5% label noise makes over half of instances subject to the lottery of mistakes in the clinical tasks.
Limitations
- Hmm
Experimental setup
Datasets: shock_eicu shock_mimic mortality support lungcancer enhancer
Benchmarks: shock_eicu shock_mimic mortality support lungcancer enhancer
Metrics: true error anticipated error regret overreliance susceptibility selective error hit rate
Baselines: Ignore (standard ERM) Hedge (noise-corrected ERM) confidence-based abstention with predicted probability
Builds on: noise-tolerant ERM hedging against label noise model multiplicity
Closest papers by meaning
- Understanding Instance-Level Label Noise: Disparate Impacts and Treatments (ICML 2021 · similarity 0.88)
- Learning from Noisy Labels via Conditional Distributionally Robust Optimization (NeurIPS 2024 · similarity 0.85)
- Label Noise: Ignorance Is Bliss (NeurIPS 2024 · similarity 0.85)
- Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimization (NeurIPS 2025 · similarity 0.85)
- Understanding Domain Generalization: A Noise Robustness Perspective (ICLR 2024 · similarity 0.85)
- Probably Approximately Precision and Recall Learning (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.