Research map / Label-efficient and robust learning / OOD and anomaly detection
ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ImageNet-OOD, a manually curated 31,807-image semantic-shift OOD dataset built from ImageNet-21K to minimize covariate shift, and shows modern OOD detectors respond more to covariate shift and offer little gain over MSP.
Problem
OOD detection evaluation conflates semantic shift with covariate shift, since existing ImageNet-based semantic OOD datasets contain ID contamination, semantic/visual ambiguity, and unnecessary covariate shifts; it is unclear what modern OOD detectors actually detect.
Approach
Construct a clean semantic-shift OOD benchmark (ImageNet-OOD) from ImageNet-21K by removing hypernyms/hyponyms of ID classes, the organism subtree, and visually ambiguous classes, then use it with covariate-shift datasets to decouple the two shift types and analyze detectors under new-class and failure detection.
Key results
- ImageNet-OOD contains 31,807 images from 637 classes; on ImageNet-1K vs ImageNet-OOD, the best detector (Max-Cosine/ASH-B) improves over MSP by under 1% AUROC (Max-Logit +0.7%) for new-class detection.
- On ImageNet-R vs ImageNet-OOD, MSP achieves 48.0 AUROC while ViM achieves 35.9 and ReAct achieves 35.0 for new-class detection, all below 50.
- For failure detection on ImageNet-1K vs ImageNet-OOD, MSP achieves 86.9 AUROC, outperforming Max-Logit (84.1) and ViM (82.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Max-Logit instead of MSP | ImageNet-1K OOD detection · ImageNet-1K vs. ImageNet-OOD (new-class detection) | AUROC: 79.8 -> 80.5 | Helped |
| Use Max-Logit instead of MSP | ImageNet-1K OOD detection · ImageNet-1K vs. ImageNet-OOD (failure detection) | AUROC: 86.9 -> 84.1 | Hurt |
| Use Max-Logit instead of MSP | ImageNet-1K OOD detection · ImageNet-1K vs. OpenImage-O (new-class detection) | AUROC: 84.6 -> 87.9 | Helped |
| Use ViM instead of MSP | ImageNet-1K OOD detection · ImageNet-1K vs. OpenImage-O (new-class detection) | AUROC: 84.6 -> 88.9 | Helped |
| Use ASH-B instead of MSP | ImageNet-1K OOD detection · ImageNet-1K vs. OpenImage-O (new-class detection) | AUROC: 84.6 -> 90.1 | Helped |
| Use ViM instead of MSP | ImageNet-1K OOD detection · ImageNet-1K vs. ImageNet-OOD (new-class detection) | AUROC: 79.8 -> 79.8 | No clear effect |
| Use ASH-B instead of MSP | ImageNet-1K OOD detection · ImageNet-1K vs. ImageNet-OOD (new-class detection) | AUROC: 79.8 -> 79.5 | Hurt |
| Use modern OOD detectors instead of MSP | OOD detection across covariate-shift datasets vs. ImageNet-OOD · ImageNet-C, ImageNet-R, and ImageNet-Sketch vs. ImageNet-OOD | AUROC | Hurt |
| Use modern detectors rather than MSP | Failure detection · ImageNet-1K vs. ImageNet-OOD | AUROC: 86.9 vs. 84.1 | Hurt |
| Apply a zoom transformation to the ImageNet-1K ostrich image | ResNet-50 OOD detector scores · ImageNet-OOD score ranking | rank above ImageNet-OOD samples: 99% -> 83.7%, 61.6%, and 61.4% | Hurt |
| Use covariate-shift examples rather than ImageNet-OOD examples at similar PASS distances | Energy, ViM, and ReAct · ImageNet-R vs. ImageNet-OOD | AUROC | Helped |
| Use a random untrained model instead of a trained model | OOD detectors · ImageNet-OOD | AUROC: 0.5 | No clear effect |
Surprising findings
- Modern OOD detectors (e.g., ViM, Energy, ReAct) rank correctly classified covariate-shifted images as more OOD-like than semantic-shift images, and detectors fail a random-model sanity check.
- On ImageNet-1K vs ImageNet-OOD, MSP outperforms Max-Logit by 2.8% on failure detection despite Max-Logit winning by 0.7% on new-class detection.
Limitations
- Hyperparameters for detectors taken from original authors' ImageNet-1K ablations rather than tuned on ImageNet-OOD
- Evaluation restricted to ImageNet-1K-trained torchvision CNNs
- Manual curation required about 26 hours of human effort, limiting scalability
Experimental setup
Datasets: ImageNet-1K ImageNet-21K ImageNet-C ImageNet-R ImageNet-Sketch OpenImage-O PASS
Benchmarks: ImageNet-OOD OpenImage-O ImageNet-C ImageNet-R ImageNet-Sketch
Metrics: AUROC
Baselines: MSP Max-Logit Energy Mahalanobis ViM KNN Max-Cosine ASH-B ReAct
Builds on: MSP ViM ReAct Energy PASS
Closest papers by meaning
- Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox (NeurIPS 2024 · similarity 0.94)
- The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection (NeurIPS 2024 · similarity 0.91)
- A Systematic Analysis of Out-of-Distribution Detection Under Representation and Training Paradigm Shifts (NeurIPS 2026 · similarity 0.90)
- Image Background Serves as Good Proxy for Out-of-distribution Data (ICLR 2024 · similarity 0.90)
- Expecting The Unexpected: Towards Broad Out-Of-Distribution Detection (NeurIPS 2024 · similarity 0.90)
- MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.