Research map / Label-efficient and robust learning / OOD and anomaly detection
Position: Supervised Classifiers Answer the Wrong Questions for OOD Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that OOD detection methods built on supervised classifiers trained on in-distribution data answer the wrong question, illustrated with feature- and logit-based failure analyses and tests of common interventions.
Problem
Methods that detect out-of-distribution inputs using the uncertainty, logits, or features of a classifier trained only on in-distribution classes are misspecified for the OOD question, leading to irreducible errors.
Approach
Argues that feature-based (distance) and logit-based (uncertainty) OOD scores answer different questions than "is this input from a different distribution?", and that common fixes (hybrid scores, outlier exposure, epistemic uncertainty, extra OOD class, generative models, scaling) do not resolve this misalignment.
Key results
- Oracle binary classifier on ViT-S/16 features shows irreducible error; Mahalanobis with oracle PCA features improves AUROC by over 10 points on average across models.
- MSP, max-logit, energy, and entropy average FPR@95 of about 66.5% across 14 ImageNet-trained models and three OOD datasets (Table A.1).
- Outlier exposure reduces covariate-shift accuracy by over 10% on CIFAR-10 benchmarks compared to standard ERM training with ResNet-18.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Mahalanobis distance on supervised features to detect dogs as OOD | ResNet-18 trained on airplane, cat, and truck classes · CIFAR-10 subset vs dog | AUROC | Hurt |
| Use MSP, Max-Logit, or Energy Score for OOD scoring | ResNet-50 trained on ImageNet-1K · ImageNet-1K vs Textures (striped) | AUROC: MSP: 0.484; Max-Logit: 0.654; Energy Score: 0.665 | Hurt |
| Select only OOD-discriminative features using oracle PCA instead of all features (ablation) | Mahalanobis OOD detection · ImageNet-1K vs IN-OOD, Textures, and iNaturalist | AUROC: over 10 percentage points | Helped |
| Choose PCA features for one OOD dataset and apply them to other OOD datasets (ablation) | Mahalanobis OOD detection with ViT-S/16 features · ImageNet-1K vs IN-OOD, Textures, and iNaturalist | AUROC | Mixed |
| Combine normalized Mahalanobis and MSP scores with Hybrid-Add | Supervised OOD detection methods · Textures, iNaturalist, and IN-OOD | AUROC | Mixed |
| Add outlier exposure using TIN-597 during training | ResNet-18 trained on CIFAR-10 · CIFAR-C and other covariate shifts | accuracy: over 10% | Hurt |
| Increase training data from 30 to 10000 points for Bayesian last-layer Laplace approximation (ablation) | ResNet-18 features with Bayesian last layer · CIFAR-100 OOD detection (three-class task) | AUROC: 0.769 -> 0.741 | Hurt |
| Use MAP estimates instead of Bayesian Laplace approximation (ablation) | ResNet-18 features with last-layer classifier · CIFAR-100 OOD detection (three-class task) | AUROC: 0.741 -> 0.735 | Hurt |
| Train an OOD class using examples from cup and skyscraper | ResNet-18 trained on two CIFAR-100 ID classes · Remaining CIFAR-100 classes | OOD detection performance | Mixed |
| Use diffusion loss from a DiT trained on ImageNet-1K for OOD scoring | DiT · ImageNet-1K vs Textures and iNaturalist | AUROC: 0.462; 0.694 | Mixed |
| Increase model and pretraining scale | Supervised feature-based OOD detection · ImageNet-1K vs ImageNet-OOD | AUROC: over 5% irreducible error | Helped |
Surprising findings
- Better generative models of in-distribution data (e.g., GMM with covariance interpolated toward identity, or RealNVP with higher likelihood) can yield worse OOD detection.
- Epistemic uncertainty via Laplace approximation becomes less useful for OOD detection as more ID data is observed due to posterior collapse.
Limitations
- Empirical evidence relies on specific benchmark OOD datasets and small-scale models (e.g., LeNet-5, ResNet-18 on CIFAR-10).
- Position paper; the proposed alternative (directly estimating the probability an input comes from a different distribution) is not developed or evaluated.
Experimental setup
Datasets: ImageNet-1k CIFAR-10 CIFAR-100 ImageNet-OOD Textures iNaturalist TIN-597 CelebA STL-9 CINIC-10 CIFAR-C MNIST
Benchmarks: ImageNet-OOD Textures iNaturalist CIFAR-10 vs CIFAR-100 CelebA vs CIFAR-10 CIFAR-C
Metrics: AUROC FPR@95 accuracy
Baselines: MSP Mahalanobis distance Relative Mahalanobis ViM Max Logit Energy score Entropy Outlier Exposure
Builds on: maximum softmax probability Mahalanobis distance ViM outlier exposure normalizing flows
Closest papers by meaning
- The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection (NeurIPS 2024 · similarity 0.90)
- A Noisy Elephant in the Room: Is Your Out-of-Distribution Detector Robust to Label Noise? (CVPR 2024 · similarity 0.90)
- Expecting The Unexpected: Towards Broad Out-Of-Distribution Detection (NeurIPS 2024 · similarity 0.90)
- Spurious-Aware Prototype Refinement for Reliable Out-of-Distribution Detection (NeurIPS 2025 · similarity 0.90)
- Can We Ignore Labels in Out of Distribution Detection? (ICLR 2025 · similarity 0.89)
- Long-Tailed Out-of-Distribution Detection via Normalized Outlier Distribution Adaptation (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.