Research map / Label-efficient and robust learning / Noisy labels

Enhancing Noise-Robust Losses for Large-Scale Noisy Data Learning

AAAI 2025 · Label-efficient and robust learning › Noisy labels · approach: Sample selection and co-training · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes "logit bias," adding a class-count-dependent constant to the correct-class logit so bounded noise-robust losses (especially MAE) get non-vanishing gradients at initialization, and calculates loss hyperparameters without training.

Problem

Bounded noise-robust losses such as MAE underfit multi-class datasets because, at initialization, network logits barely overlap with the region where the loss derivative is non-vanishing, and this overlap shrinks as class count grows.

Approach

Analyze the average backpropagation error of the correct-class neuron against the initial logit distribution N(0,1); shift the correct-class logit by a class-count-dependent bias epsilon (MAE*) so that gradients are non-zero at init, and use the same error-matching criterion to set hyperparameters of genCE, NCE-AGCE and NF-MAE without training.

Key results

• On Cifar-10 with 0% noise, MAE* (epsilon=0.5) reaches 91.27% vs 84.90% for MAE and 91.30% for CE, with MAE* best at 20-40% symmetric noise.
• On WebVision (1000 classes), only MAE* among the noise-robust losses learns from scratch: genCE+ reaches 64.85% top-1 and MAE** reaches 81.11% top-5, while plain genCE, NF-MAE and NCE-AGCE stay near chance at 1-13% top-1.
• On Cifar-100 with 0% noise, MAE* (epsilon=3.0) reaches 76.49%, the best of all losses, versus 6.84% for MAE.

Measured effects reported in the paper

Surprising findings

• Plain MAE fails on Cifar-100 (about 6.8% accuracy in the noise-free case), yet the logit bias alone lifts it to about 76%, the best result among the compared losses.
• Every noise-robust loss tested that works on Cifar-10 and Cifar-100 fails to learn WebVision without modification, except MAE*.

Limitations

• The logit bias formula tends to overestimate the required epsilon, which can compromise noise robustness at high noise levels (e.g., epsilon=3.0 overfits at 40-60% noise).
• Scope is limited to image classification.
• On WebVision, large epsilon makes test-time confidence low, requiring a hand-designed epsilon schedule that the authors consider a form of tuning.

Experimental setup

Datasets: Cifar-10 Cifar-100 Fashion-MNIST WebVision

Benchmarks: Cifar-10 Cifar-100 Fashion-MNIST WebVision

Metrics: test accuracy top-1 accuracy top-5 accuracy

Baselines: CE MAE genCE AGCE NF-MAE NCE-MAE NCE-AGCE biTemp symCE

Builds on: MAE loss generalized cross entropy normalized cross entropy active-passive losses bi-tempered logistic loss

Closest papers by meaning

• Normalized Loss Functions for Deep Learning with Noisy Labels (ICML 2020 · similarity 0.91)
• Asymmetric Loss Functions for Learning with Noisy Labels (ICML 2021 · similarity 0.91)
• Robust Loss Functions under Label Noise for Deep Neural Networks (AAAI 2017 · similarity 0.90)
• $\epsilon$-Softmax: Approximating One-Hot Vectors for Mitigating Label Noise (NeurIPS 2024 · similarity 0.89)
• Robust Loss Functions for Training Decision Trees with Noisy Labels (AAAI 2024 · similarity 0.87)
• Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimization (NeurIPS 2025 · similarity 0.87)