Research map / Label-efficient and robust learning / OOD and anomaly detection
Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds IS-OOD, an OOD detection benchmark that splits ImageNet-21K test samples into 8x8 semantic/covariate shift levels using a CLIP-based decomposition (LAID), plus a generated Syn-IS dataset from SDXL-Turbo, and evaluates eight OOD detectors on ResNet-50.
Problem
Standard OOD detection benchmarks label any sample with a novel semantic label as OOD, but marginal samples are semantically close to ID data because labels are inaccurate or incomplete, making the OOD boundary a Sorites paradox; covariate shifts are also not separated from semantic shifts.
Approach
Instead of a binary ID/OOD split by label, measure each test sample's semantic and covariate shift degree relative to ImageNet-1K using an orthogonal transformation of CLIP features trained with triplet and orthogonality losses on text, then bin samples into incremental shift levels; complement with generated style-diverse images.
Key results
- Most OOD detectors (e.g., MSP, KNN) gain nearly 40% AUROC increase from smallest to largest semantic shift, except GradNorm and RankFeat.
- On ImageNet-21K, semantic correlation is positive (e.g., KNN 0.98, sensitivity 6.64) while covariate correlation is negative for most methods; on Syn-IS, several methods show positive covariate correlation (e.g., MDS 0.99).
- On the labeling-issue analysis, MSP mean AUROC rises from 68.0 on the full ImageNet-21K to 79.5 on subsets with larger semantic shifts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate on subsets with larger semantic shifts and fewer labeling issues rather than the full ImageNet-21K set | OOD detection methods · ImageNet-21K | AUROC: MSP: 68.0 -> 79.5; ODIN: 74.1 -> 84.9; MDS: 52.3 -> 58.9; GradNorm: 70.1 -> 73.0; KNN: 65.4 -> 78.7; DICE: 69.4 -> 78.9; | Helped |
| Increase semantic shift level | Most OOD detection methods · ImageNet-21K | AUROC: nearly 40% increase in AUROC under the largest semantic shift compared to the result under the smallest semantic shift | Helped |
| Increase covariate shift level | OOD detection methods · ImageNet-21K | AUROC | Mixed |
| Increase semantic shift level | GradNorm and RankFeat · ImageNet-21K | AUROC | No clear effect |
| Use highly shifted synthetic styles as covariate content | OOD detection methods · Syn-IS | AUROC | Helped |
| Evaluate on generated images instead of ImageNet-21K real images | GradNorm · Syn-IS | AUROC | Hurt |
| Use LAID semantic features instead of full CLIP features | CLIP zero-shot classifier · ImageNet-R | Acc@1: 64.82 -> 66.81 | Helped |
| Use PCA semantic features instead of full CLIP features | CLIP zero-shot classifier · ImageNet-1K | Acc@1: 70.11 -> 39.23 | Hurt |
Surprising findings
- GradNorm and RankFeat show little AUROC change with semantic shift, suggesting they rely on different cues than semantic novelty.
- Excessive covariate shifts (e.g., papercut styles) on Syn-IS are often flagged as OOD by several methods, increasing their performance.
Limitations
- CLIP text and image feature spaces are not perfectly aligned, so the decomposition matrix may not transfer exactly to images.
- Covariate shifts are limited to styles, augmentations, and lighting; semantic and covariate factors are correlated in ImageNet-21K, leaving some subsets empty.
- Evaluation is limited to eight OOD detectors on a single ResNet-50 ImageNet-1K classifier.
Experimental setup
Datasets: ImageNet-1K ImageNet-21K ImageNet-C ImageNet-R PACS NINCO
Benchmarks: IS-OOD (ImageNet-21K subsets) Syn-IS
Metrics: AUROC FPR@95 AUPR Pearson correlation of performance with shift level sensitivity to shift
Baselines: MSP ODIN MDS GradNorm KNN DICE RankFeat ASH
Builds on: CLIP OpenOOD SDXL SDXL-Turbo ImageNet-21K
Closest papers by meaning
- ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms (ICLR 2024 · similarity 0.94)
- Image Background Serves as Good Proxy for Out-of-distribution Data (ICLR 2024 · similarity 0.92)
- A Systematic Analysis of Out-of-Distribution Detection Under Representation and Training Paradigm Shifts (NeurIPS 2026 · similarity 0.91)
- The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection (NeurIPS 2024 · similarity 0.91)
- MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities (NeurIPS 2024 · similarity 0.90)
- EAT: Towards Long-Tailed Out-of-Distribution Detection (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.