Research map / Vision-language and multimodal / CLIP and zero-shot
Negative Label Guided OOD Detection with Pretrained Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NegLabel, a post hoc zero-shot OOD detector for CLIP-like VLMs that adds up to 10,000 WordNet-mined negative labels and scores OOD via a sum-softmax over ID vs. negative label similarities.
Problem
Zero-shot OOD detection with vision-language models: existing VLM-based OOD scores use only the ID label space, leaving text knowledge underexploited, and performance degrades on large ID label sets such as ImageNet-1k.
Approach
Expand the label space with a large set of negative labels mined from WordNet (NegMining selects words far from ID labels using a 5th-percentile distance), and score OOD with a sum-softmax that rewards affinity to ID labels relative to negative labels, plus a grouping strategy that averages scores over label groups.
Key results
- With CLIP-B/16 on ImageNet-1k, NegLabel reaches 94.21% average AUROC and 25.40% FPR95, versus 90.82% AUROC and 43.93% FPR95 for MCM.
- On ImageNet-1k with iNaturalist as OOD, NegLabel achieves 99.49% AUROC and 1.91% FPR95.
- NegLabel generalizes across architectures, e.g., AltCLIP ViT-L/14 at 94.60% AUROC and ALIGN EfficientNet-B7 at 90.76% AUROC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NegLabel instead of CLIPN | CLIP-B/16 · ImageNet-1k OOD (average over iNaturalist, SUN, Places, Textures) | AUROC: 93.10 -> 94.21 | Helped |
| use NegLabel instead of CLIPN | CLIP-B/16 · ImageNet-1k OOD (average over iNaturalist, SUN, Places, Textures) | FPR95: 31.10 -> 25.40 | Helped |
| add NegMining to select 10,000 negative labels (ablation) | NegLabel · ImageNet-1k OOD (average over iNaturalist, SUN, Places, Textures) | AUROC: 91.95 -> 93.76 | Helped |
| add NegMining to select 10,000 negative labels (ablation) | NegLabel · ImageNet-1k OOD (average over iNaturalist, SUN, Places, Textures) | FPR95: 34.82 -> 28.19 | Helped |
| add grouping to NegLabel with NegMining and 10,000 negative labels (ablation) | NegLabel · ImageNet-1k OOD (average over iNaturalist, SUN, Places, Textures) | AUROC: 93.76 -> 94.21; +0.45% | Helped |
| add grouping to NegLabel with NegMining and 10,000 negative labels (ablation) | NegLabel · ImageNet-1k OOD (average over iNaturalist, SUN, Places, Textures) | FPR95: 28.19 -> 25.40; -2.79% | Helped |
| add grouping to randomly selected 10,000 negative labels (ablation) | NegLabel without NegMining · ImageNet-1k OOD (average over iNaturalist, SUN, Places, Textures) | AUROC: 91.95 -> 92.01 | No clear effect |
| add grouping to randomly selected 10,000 negative labels (ablation) | NegLabel without NegMining · ImageNet-1k OOD (average over iNaturalist, SUN, Places, Textures) | FPR95: 34.82 -> 34.69 | No clear effect |
| increase NegMining-selected negative labels from 10,000 to 20,000 (ablation) | NegLabel · ImageNet-1k OOD (average over iNaturalist, SUN, Places, Textures) | AUROC: 94.21 -> 93.85 | Hurt |
| increase NegMining-selected negative labels from 10,000 to 20,000 (ablation) | NegLabel · ImageNet-1k OOD (average over iNaturalist, SUN, Places, Textures) | FPR95: 25.40 -> 26.56 | Hurt |
| increase NegMining-selected negative labels from 10,000 to 50,000 (ablation) | NegLabel · ImageNet-1k OOD (average over iNaturalist, SUN, Places, Textures) | AUROC: 94.21 -> 92.70 | Hurt |
| increase NegMining-selected negative labels from 10,000 to 50,000 (ablation) | NegLabel · ImageNet-1k OOD (average over iNaturalist, SUN, Places, Textures) | FPR95: 25.40 -> 31.67 | Hurt |
Surprising findings
- Randomly selected WordNet negative labels perform better than NegMining when the number of negative labels is below 1,000.
- Using all WordNet words as negative labels still outperforms previous methods on ImageNet-1k.
Limitations
- Performance peaks at around 10,000 negative labels and declines with more, since added labels become dissimilar and less informative.
- Theoretical analysis relies on an i.i.d. assumption over negative labels that the authors acknowledge may not hold.
- Hyperparameters such as the percentile, temperature and group count are tuned on the ImageNet-1k benchmark.
Experimental setup
Datasets: WordNet ImageNet-1k
Benchmarks: ImageNet-1k OOD benchmark (iNaturalist, SUN, Places, Textures) ImageNet-10/20/100 hard OOD ImageNet-Sketch, ImageNet-A, ImageNet-R, ImageNetV2 domain shift Stanford-Cars, CUB-200, Oxford-Pet, Food-101 as ID
Metrics: AUROC FPR95
Baselines: MCM ZOC CLIPN Energy Mahalanobis MSP ODIN NPOS KNN VOS ViM GradNorm
Builds on: MCM CLIP NPOS
Closest papers by meaning
- Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Learning Transferable Negative Prompts for Out-of-Distribution Detection (CVPR 2024 · similarity 0.88)
- Can We Ignore Labels in Out of Distribution Detection? (ICLR 2025 · similarity 0.88)
- Envisioning Outlier Exposure by Large Language Models for Out-of-Distribution Detection (ICML 2024 · similarity 0.88)
- A Noisy Elephant in the Room: Is Your Out-of-Distribution Detector Robust to Label Noise? (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.