Research map / Label-efficient and robust learning / Noisy labels

Rejection via Learning Density Ratios

NeurIPS 2024 · Label-efficient and robust learning › Noisy labels · approach: Contrastive learning on noisy data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes rejection via density ratios between a learned idealized distribution (phi-divergence-regularized risk minimization) and the data distribution, with closed forms for KL and alpha-divergences, evaluated on six datasets with and without label noise.

Problem

Classification with rejection usually requires training a new model with augmented losses; post-hoc rejection on pretrained classifiers is less explored, and existing approaches need probability estimates or retraining.

Approach

Find an idealized data distribution that minimizes a model's expected loss regularized by a phi-divergence from the data distribution, then reject inputs whose density ratio to the data distribution falls below a threshold; closed forms are derived for KL and alpha-divergences (including chi-squared), linking to Chow's rule, DRO and GVI.

Key results

• At 80% coverage on clean CIFAR-10, KL-Rej reaches 97.22% accuracy versus 90.20% for the base model and 95.45% for CSS.
• On 25% noisy CIFAR-10, the alpha=3 rejector reaches 92.50% at 81% coverage, versus 85.31% for the base model.
• Density-ratio rejectors are competitive with or superior to the baselines on most datasets, while PredRej is outperformed by them in the reported table.

Measured effects reported in the paper

Surprising findings

• KL and alpha=3 density-ratio rejectors differ mainly in the coverage range covered by the threshold, and perform similarly otherwise.
• Baselines that train with rejection simultaneously are weaker at coverage above 95% than wrapping approaches, which the authors attribute to the lack of architecture tuning.

Limitations

• Relies on the base model's calibrated probability estimates to approximate P(Y|X), which is a limitation acknowledged by the authors.
• Closed-form solutions are only available for KL and alpha>1 (chi-squared given bounded loss); general phi-divergences require difficult Lagrange multiplier solving.
• Test set rejection rates depend on tuning tau on a validation set.

Experimental setup

Datasets: HAR Gas Drift MNIST CIFAR-10 OrganMNIST OctMNIST

Benchmarks: HAR Gas Drift MNIST CIFAR-10 OrganMNIST OctMNIST

Metrics: accuracy acceptance coverage

Baselines: DEFER GCE CSS PredRej

Builds on: Generalized Variational Inference Distributionally Robust Optimization Chow's rule temperature scaling

Closest papers by meaning

• On discriminative learning of prediction uncertainty (ICML 2019 · similarity 0.86)
• Density-Ratio Losses for Post-Hoc Learning to Defer (NeurIPS 2026 · similarity 0.85)
• Overcoming Saturation in Density Ratio Estimation by Iterated Regularization (ICML 2024 · similarity 0.83)
• Learning to Reject with a Fixed Predictor: Application to Decontextualization (ICLR 2024 · similarity 0.83)
• DRoP: Distributionally Robust Data Pruning (ICLR 2025 · similarity 0.83)
• When majority rules, minority loses: bias amplification of gradient descent (NeurIPS 2025 · similarity 0.83)