Research map / Learning theory and algorithms / Prediction and decision losses
Robust Loss Functions for Training Decision Trees with Noisy Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes robustness of loss functions for decision tree learning under label noise, introduces conservative losses and a distribution-loss framework, and proposes the adaptive negative exponential loss, which gives an efficient impurity for greedy tree splitting.
Problem
Decision tree learning from noisily labeled data is sensitive to label noise, and little is known about which loss functions (and their induced impurities) are robust in the tree setting.
Approach
Defines conservative losses (whose minimum partial risk equals the misclassification impurity, yielding noise-tolerant predictions and early stopping), a distribution-loss framework ell = F(-margin) for CDF F, and the negative exponential loss whose impurity is min{1-||p||_inf, lambda*sqrt((1-||p||_2^2)/2)}, with lambda tunable to noise rate.
Key results
- Adaptive negative exponential loss (ANE) is almost always a top performer across noise settings, label configurations and datasets for decision trees, e.g., MNIST binary with uniform noise 0.4: 82.03 ± 1.39 vs CE 58.19 ± 0.65.
- Conservative losses give strong performance under high noise but sometimes poor performance in low-noise settings.
- Differences across loss functions are smaller for random forests than for single decision trees.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive negative exponential loss (tune λ) instead of the other evaluated splitting criteria | Decision trees trained on noisy labels · Multiple datasets and noise settings | test accuracy | Helped |
| Use conservative losses rather than non-conservative losses | Decision trees trained on noisy labels · High label noise | test accuracy | Helped |
| Use conservative losses rather than non-conservative losses | Decision trees trained on noisy labels · Low label noise | test accuracy | Hurt |
| Use twoing split criteria instead of the other evaluated criteria | Decision trees trained on noisy labels · Multiple datasets and noise settings | test accuracy | Hurt |
| Use ANE instead of other splitting criteria | Random forests trained on noisy labels · Binary and multiclass settings | test accuracy | Helped |
| Use random forest ensembles rather than a single decision tree | Tree methods trained on noisy labels · Multiple noise settings | test accuracy | Helped |
| Tune the NE robustness parameter λ for each noise setting (ablation) | NE decision tree · MNIST digits | test accuracy | Mixed |
| Vary GCE q between 0 and 1 (ablation) | GCE tree methods · Label-noise robustness | test accuracy | No clear effect |
| Use pre-trained VGG11 image embeddings instead of raw CIFAR-10 pixels (ablation) | Tree methods · CIFAR-10 | test accuracy | Helped |
Surprising findings
- Conservative losses can underfit in low-noise settings, since early stopping halts splitting too soon.
- Varying GCE q between 0 and 1 does not improve tree robustness to label noise.
Limitations
- Capped NE loss has zero gradient, so it cannot be used for gradient-based learning (only for tree impurity).
- Conservative losses underfit in low-noise settings.
- Only class-conditional and uniform noise are considered.
Experimental setup
Datasets: MNIST CIFAR-10 20News UNSW-NB15 Covertype Mushrooms
Benchmarks: MNIST CIFAR-10 20News UNSW-NB15 Covertype Mushrooms
Metrics: test accuracy 2x standard deviation bands
Baselines: Twoing split criterion Credal-C4.5 GCE loss ranking loss cross entropy (CE) mean squared error (MSE) conservative losses (CL)
Builds on: GCE loss ranking loss impurity CART impurity-reduction view
Closest papers by meaning
- Popular decision tree algorithms are provably noise tolerant (ICML 2022 · similarity 0.89)
- Variation-Bounded Loss for Noise-Tolerant Learning (AAAI 2026 · similarity 0.87)
- Toward a Characterization of Loss Functions for Distribution Learning (NeurIPS 2019 · similarity 0.87)
- Asymmetric Loss Functions for Learning with Noisy Labels (ICML 2021 · similarity 0.87)
- Enhancing Noise-Robust Losses for Large-Scale Noisy Data Learning (AAAI 2025 · similarity 0.87)
- Learning Small Decision Trees with Few Outliers: A Parameterized Perspective (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.