Research map / Label-efficient and robust learning / OOD and anomaly detection
In or Out? Fixing ImageNet Out-of-Distribution Detection Evaluation
Julian Bitterwolf, Maximilian Müller, Matthias Hein
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace MSP with Mahalanobis distance | ViT-B-384 pretrained on IN-21K · NINCO | mean FPR: 51.9 -> 27.5 | Improved |
| Replace MSP with Relative Mahalanobis | ViT-B-384 pretrained on IN-21K · NINCO | mean FPR: 51.9 -> 31.2 | Improved |
| Replace MSP with KNN | ViT-B-384 pretrained on IN-21K · NINCO | mean FPR: 51.9 -> 62.7 | Hurt |
| Replace MSP with Relative Mahalanobis | ViT-B-384 without pretraining · NINCO | mean FPR: 69.5 -> 57.1 | Improved |
| Use CLIP plus IN-12K pretraining rather than no pretraining | ViT-B-384 with MSP · NINCO | mean FPR: 69.5 -> 50.2 | Improved |
| Remove ID-contaminated samples from existing OOD test datasets | MSP and Mahalanobis OOD evaluation · Cleaned subsets of four popular OOD datasets | FPR at 95% TPR: 25.6% gain of MSP vs. 33.2% gain of Mahalanobis on Species; 19.6% gain vs. 26.3% gain on Places | Improved |
| Use ViM instead of MSP | ViT pretrained on IN-21K · OOD unit-tests: white | FPR: 0.0 -> 100.0 | Hurt |
| Use ViM instead of MSP | ConvNeXt without pretraining · OOD unit-tests: black | FPR: 0.0 -> 100.0 | Hurt |
| Use feature-based OOD methods, including Mahalanobis, Relative Mahalanobis, and ViM, rather than MSP | OOD detectors on NINCO · NINCO | mean FPR | Mixed |
| Use zero-shot CLIP methods instead of IN-1K classifiers | CLIP zero-shot models · NINCO | mean FPR | Hurt |
| Use OOD unit-tests alongside natural OOD data | OOD detector evaluation · OOD unit-tests | failed unit-tests | Mixed |
Closest papers by meaning
- Enhancing the Power of OOD Detection via Sample-Aware Model Selection (CVPR 2024 · similarity 0.92)
- Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox (NeurIPS 2024 · similarity 0.91)
- Image Background Serves as Good Proxy for Out-of-distribution Data (ICLR 2024 · similarity 0.91)
- OODD: Test-time Out-of-Distribution Detection with Dynamic Dictionary (CVPR 2025 · similarity 0.91)
- Test-Time Linear Out-of-Distribution Detection (CVPR 2024 · similarity 0.91)
- ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
