Research map / Label-efficient and robust learning / OOD and anomaly detection
A Noisy Elephant in the Room: Is Your Out-of-Distribution Detector Robust to Label Noise?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks 20 post-hoc OOD detectors (from OpenOOD) on classifiers trained with real and synthetic label noise across CIFAR-10/100, Clothing1M, and CUB/Aircraft, finding that noisy labels degrade OOD detection and that most methods poorly separate misclassified ID samples from OOD samples.
Problem
Post-hoc out-of-distribution detection is studied almost exclusively with clean, curated ID training labels; how it behaves when the classifier is trained on unreliable (noisy) labels is unknown.
Approach
Systematically evaluates OOD detectors under real crowd-sourced/web-scraped label noise and matched synthetic uniform and class-conditional noise, introducing AUROC metrics that separately measure correct-vs-OOD and incorrect-vs-OOD separation to show that poor incorrect-ID-vs-OOD separation is the key bottleneck.
Key results
- Median AUROC ID vs OOD drops by over 5% on CIFAR-10 with only ~9% incorrect labels (CIFAR-10N-Agg).
- Only GRAM, KNN, MDS, MDSEnsemble and VIM reach 90%+ AUROC on at least one noisy dataset.
- For all methods, AUROC incorrect vs OOD is weakly or not correlated with accuracy (rho < 0.2), and MSP's median AUROC incorrect vs OOD is around 0.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the classifier with noisy rather than clean labels | Post-hoc OOD detectors · CIFAR-10N-Agg | Median AUROC ID vs. OOD: over 5% | Hurt |
| Use synthetic uniform noise rather than real instance-dependent label noise | Post-hoc OOD detectors · CIFAR-10, CIFAR-100, and Clothing1M | AUROC ID vs. OOD | Hurt |
| Use feature-space distance-based scoring rather than class logits or probabilities | Post-hoc OOD detectors · Noisy-label settings | AUROC ID vs. OOD | Helped |
| Use ID accuracy as a proxy for OOD detection performance | Post-hoc OOD detectors · Noisy-label settings | AUROC incorrect ID vs. OOD: ρ < 0.2 | No clear effect |
| Use synthetic uniform noise rather than class-conditional noise | Post-hoc OOD detectors · Matched-noise-rate datasets | AUROC ID vs. OOD | Hurt |
| Early-stop classifier training rather than use the last checkpoint | Post-hoc OOD detectors · Noisy-label settings | AUROC ID vs. OOD | Helped |
| Tune temperature using ID validation labels rather than use the MSP baseline | TempScale · Noisy-label settings | AUROC ID vs. OOD: ϵmin[MSP>TempScale] = 0.15 | Hurt |
| Tune ODIN temperature using OOD validation detection performance rather than MSP | ODINnopert · Noisy-label settings | AUROC ID vs. OOD: ϵmin[ODINnopert>MSP] = 0.05 | Helped |
| Add input perturbation to ODIN (ablation) | ODIN · Noisy-label settings | AUROC ID vs. OOD | No clear effect |
| Use noisy training labels instead of clean validation labels to compute Mahalanobis class-wise statistics | MDS and RMDS · Noisy-label settings | AUROC ID vs. OOD: ϵmin[MDStrain>MDSval] = 0.19; ϵmin[RMDStrain>RMDSval] = 0 | Helped |
| Use noisy training labels instead of clean validation labels to set up OpenMax | OpenMax · Noisy-label settings | AUROC ID vs. OOD: ϵmin[OpenMaxtrain>OpenMaxval] = 0 | Helped |
| Use clean validation samples instead of noisy training samples to compute class-specific feature bounds | GRAM · Noisy-label settings | AUROC ID vs. OOD: ϵmin[GRAMval>GRAMtrain] = 0.23 | Helped |
Surprising findings
- Uniform synthetic label noise degrades OOD detection more than real or class-conditional noise at the same overall noise rate.
- Early stopping improves OOD detection for most methods but can accelerate OOD performance drop per unit of accuracy loss, e.g. for TempScaling.
Limitations
- Only post-hoc OOD detectors and standard discriminative training were studied; training-based OOD methods were not evaluated.
- Architectures are lightweight (ResNet18, MLP-Mixer, CCT), and results for distance-based methods vary strongly with architecture.
- Clothing1M subset is small and the realistic pre-trained setting is limited to CUB and FGVC-Aircraft with synthetic noise only.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 CIFAR-10N CIFAR-100N Clothing1M MNIST SVHN Textures EuroSAT-RGB Food-101 Stanford Online Products ImageNet
Benchmarks: CIFAR-10 CIFAR-100 Clothing1M CUB FGVC-Aircraft
Metrics: AUROC ID vs OOD AUROC correct vs OOD AUROC incorrect vs OOD Spearman correlation ASO epsilon_min
Baselines: MSP ODIN MLS EBO MDS GRAM KNN ASH
Builds on: OpenOOD CIFAR-N Almost Stochastic Order test
Closest papers by meaning
- Can We Ignore Labels in Out of Distribution Detection? (ICLR 2025 · similarity 0.92)
- When and How Does In-Distribution Label Help Out-of-Distribution Detection? (ICML 2024 · similarity 0.91)
- Position: Supervised Classifiers Answer the Wrong Questions for OOD Detection (ICML 2025 · similarity 0.90)
- EAT: Towards Long-Tailed Out-of-Distribution Detection (AAAI 2024 · similarity 0.90)
- How Does Unlabeled Data Provably Help Out-of-Distribution Detection? (ICLR 2024 · similarity 0.90)
- YolOOD: Utilizing Object Detection Concepts for Multi-Label Out-of-Distribution Detection (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.