Research map / Label-efficient and robust learning / OOD and anomaly detection
Mysteries of the Deep: Role of Intermediate Representations in Out of Distribution Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows intermediate layers of pretrained vision models carry complementary OOD signals and proposes a training-free, entropy-selected multi-layer fusion extension of MCM for zero-shot OOD detection.
Problem
Zero-shot OOD detection with pretrained vision-language and vision models typically relies only on final-layer representations, leaving intermediate-layer information unused.
Approach
Project intermediate-layer features of CLIP-style encoders into the shared text space, score each layer with MCM, and fuse layer subsets chosen by the lowest entropy of the fused-score histogram computed on unlabeled ID data, requiring no OOD data or training.
Key results
- On ImageNet-1K with ViT-B/16, the method reaches average FPR95 30.70% and AUROC 93.54, versus 42.27% and 90.83 for MCM.
- On Pascal-VOC with ViT-B/16, average FPR95 26.97% and AUROC 93.52, outperforming MCM (40.20%, 92.66).
- On near-OOD benchmarks (average of NINCO and SSB-Hard), AUROC 75.71 and FPR95 74.77, versus 68.12 and 85.08 for MCM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fuse entropy-selected intermediate layers rather than using the final layer alone | CLIP ViT-B/16 with MCM · ImageNet-1K ID, average across OOD datasets | FPR@95: 42.27 -> 30.70 | Helped |
| Fuse entropy-selected intermediate layers rather than using the final layer alone | CLIP ViT-B/16 with MCM · Pascal-VOC ID, average across OOD datasets | FPR@95: 40.20 -> 26.97 | Helped |
| Use intermediate-layer fusion instead of the final layer | CLIP ViT-B/16 with SeTAR + GL-MCM · Pascal-VOC ID, average across OOD datasets | FPR@95: 26.74 -> 26.97 | Hurt |
| Fuse intermediate layers | CLIP ViT-B/16 · Near-OOD average | AUROC: 68.12 -> 75.71 | Helped |
| Fuse intermediate layers | MCM with CLIP RN50x4 · ImageNet-1K ID, average across OOD datasets | FPR@95: 45.16 -> 41.61 | Helped |
| Fuse intermediate layers | MCM with CLIP ViT-L/14 · ImageNet-1K ID, average across OOD datasets | FPR@95: 37.16 -> 37.09 | No clear effect |
| Add intermediate-layer fusion | NegLabel with CLIP ViT-B/16 · ImageNet-1K ID, average across OOD datasets | FPR@95: 25.40 -> 23.79 | Mixed |
| Add intermediate-layer fusion | CSP with CLIP ViT-B/16 · ImageNet-1K ID, average across OOD datasets | FPR@95: 17.51 -> 15.52 | Helped |
| Select layers by entropy instead of kurtosis, standard deviation, Gini, JSD, averaging all layers, or random selection (ablation) | CLIP ViT-B/16 intermediate-layer fusion · ImageNet-1K and Pascal-VOC | FPR@95 | Helped |
| Use random layer selection instead of entropy-based selection (ablation) | CLIP ViT-B/16 intermediate-layer fusion · ImageNet-1K and Pascal-VOC | FPR@95 | Mixed |
| Fuse 3 to 6 layers rather than shorter or longer combinations (ablation) | CLIP ViT-B/16 · ImageNet-1K and Pascal-VOC OOD benchmarks | FPR@95 | Helped |
| Use histogram bin counts between 16 and 128 (ablation) | Entropy-based layer selection · ImageNet-1K and Pascal-VOC OOD benchmarks | AUROC | No clear effect |
Surprising findings
- Single intermediate layers give at best marginal and inconsistent gains over the final layer.
- Fusing many layers can raise FPR for larger models such as CLIP ViT-L/14, and models like MAE, DINOv2, SigLIP-v2 and Perception Encoder benefit less from fusion than CLIP.
Limitations
- Less effective for models with flat or redundant representations such as MAE and Perception Encoder.
- Selected layer combinations vary substantially across ID datasets, so selection does not transfer cleanly.
- Study restricted to visual inputs.
Experimental setup
Datasets: ImageNet-1K Pascal-VOC iNaturalist SUN Places Textures ImageNet-22K MS-COCO NINCO SSB-Hard
Benchmarks: ImageNet-1K vs iNaturalist/SUN/Places/Textures Pascal-VOC vs ImageNet-22K/COCO NINCO SSB-Hard
Metrics: FPR@95 AUROC
Baselines: MCM GL-MCM SeTAR LoCoOp SeTAR+FT NegLabel CSP MaxLogit Energy
Builds on: Maximum Concept Matching CLIP SeTAR GL-MCM SVCCA
Closest papers by meaning
- Detecting Out-of-Distribution Through the Lens of Neural Collapse (CVPR 2025 · similarity 0.91)
- Splitting & Integrating: Out-of-Distribution Detection via Adversarial Gradient Attribution (ICML 2025 · similarity 0.91)
- A Systematic Analysis of Out-of-Distribution Detection Under Representation and Training Paradigm Shifts (NeurIPS 2026 · similarity 0.90)
- Expecting The Unexpected: Towards Broad Out-Of-Distribution Detection (NeurIPS 2024 · similarity 0.90)
- Out-of-Distribution Detection by Leveraging Between-Layer Transformation Smoothness (ICLR 2024 · similarity 0.89)
- NECO: NEural Collapse Based Out-of-distribution detection (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.