Research map / Label-efficient and robust learning / Noisy labels
$\epsilon$-Softmax: Approximating One-Hot Vectors for Mitigating Label Noise
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ε-softmax, a plug-in softmax modification that pushes outputs toward one-hot vectors with controllable error, and combines CE_ε with MAE to get noise-robust yet well-fitting losses.
Problem
Noisy labels degrade deep network training; symmetric robust losses are robust but underfit because the symmetric condition is overly strict.
Approach
Modify the softmax output by amplifying the maximum probability by m and renormalizing, so predictions approximate one-hot vectors with controllable error sqrt(1-1/K)/(m+1); this makes any loss approximately symmetric, and combining the resulting CE_ε with MAE restores fitting ability.
Key results
- CE_ε+MAE reaches 59.20% on CIFAR-100 with 0.4 symmetric noise and 26.30% at 0.8 symmetric noise, versus 39.31% and 7.33% for CE.
- On CIFAR-100 with 0.4 asymmetric noise, CE_ε+MAE achieves 48.51% versus 41.15% for CE.
- On CIFAR-10N Worst, CE_ε+MAE (Semi) achieves 95.12%, and CE_ε+MAE reaches 83.74% on the Worst split vs 77.69% for CE; on WebVision top-1 it reaches 71.32% vs 66.08% for CE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CEε+MAE instead of NCE+AUL | CIFAR-10 · symmetric noise 0.8 | accuracy: 57.59 ±0.84 -> 58.96 ±0.70 | Helped |
| Use CEε+MAE instead of NCE+RCE | CIFAR-10 · asymmetric noise 0.4 | accuracy: 79.43 ±0.21 -> 78.91 ±0.25 | Hurt |
| Use CEε+MAE instead of NCE+RCE | CIFAR-100 · symmetric noise 0.8 | accuracy: 24.87 ±0.52 -> 26.30 ±0.46 | Helped |
| Use CEε+MAE instead of NCE+AGCE | CIFAR-100 · asymmetric noise 0.4 | accuracy: 44.15 ±0.08 -> 48.51 ±0.36 | Helped |
| Use CEε+MAE instead of Negative-LS | CIFAR-100N · Noisy | accuracy: 58.59 ±0.98 -> 61.78 ±0.14 | Helped |
| Add SSL, sample selection, pseudo-labeling, and MixUp to CEε+MAE | CIFAR-10N · Worst | accuracy: 83.74 ±0.43 -> 95.12 ±0.10 | Helped |
| Use CEε+MAE instead of LDR-KL | WebVision · validation Top-1 | accuracy: 69.64 -> 71.32 | Helped |
| Use CEε+MAE instead of LDR-KL | ILSVRC12 · validation Top-1 | accuracy: 65.24 -> 67.20 | Helped |
| Increase m from 0 to 1e2 (ablation) | CEε+MAE · CIFAR-100 symmetric noise 0.4 | accuracy: 39.72 ±0.67 -> 48.39 ±0.53 | Helped |
| Increase m from 1e2 to 1e4 (ablation) | CEε+MAE · CIFAR-100 symmetric noise 0.4 | accuracy: 48.39 ±0.53 -> 59.20 ±0.42 | Helped |
| Increase m from 1e4 to 1e5 (ablation) | CEε+MAE · CIFAR-100 symmetric noise 0.4 | accuracy: 59.20 ±0.42 -> 54.99 ±1.05 | Hurt |
| Use m=1e4 instead of m=1e2 (ablation) | CEε+MAE · CIFAR-100 asymmetric noise 0.4 | accuracy: 48.51 ±0.36 -> 40.36 ±0.96 | Hurt |
Surprising findings
- Plain CE with ε-softmax alone underfits at large m, while adding MAE both improves clean-data fit and noise robustness, with CE_ε+MAE sometimes exceeding CE on clean CIFAR-10.
- Stricter constraints (larger m) suit symmetric noise while looser constraints suit asymmetric noise.
Limitations
- ε-softmax may slightly reduce fitting ability on clean data, requiring combination with MAE.
- Hyperparameter m and MAE weights (alpha, beta) are tuned per noise setting and dataset.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 CIFAR-10N CIFAR-100N WebVision ILSVRC12 Clothing1M
Benchmarks: CIFAR-10 CIFAR-100 CIFAR-10N CIFAR-100N WebVision ILSVRC12 Clothing1M
Metrics: test accuracy top-1 accuracy top-5 accuracy
Baselines: CE FL MAE GCE NLNL SCE NCE+MAE NCE+RCE NFL+RCE NCE+AUL NCE+AGCE NCE+AEL
Builds on: Active Passive Loss Symmetric Cross Entropy Generalized Cross Entropy Mean Absolute Error Focal loss
Closest papers by meaning
- Robust Loss Functions under Label Noise for Deep Neural Networks (AAAI 2017 · similarity 0.90)
- Enhancing Noise-Robust Losses for Large-Scale Noisy Data Learning (AAAI 2025 · similarity 0.89)
- Normalized Loss Functions for Deep Learning with Noisy Labels (ICML 2020 · similarity 0.89)
- Coresets for Robust Training of Deep Neural Networks against Noisy Labels (NeurIPS 2020 · similarity 0.89)
- Dirichlet-Based Prediction Calibration for Learning with Noisy Labels (AAAI 2024 · similarity 0.87)
- Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.