Research map / Label-efficient and robust learning / OOD and anomaly detection
Conjugated Semantic Pool Improves OOD Detection with Pre-trained Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a conjugated semantic pool (CSP) of adjective-modified superclass names to expand OOD label candidates for zero-shot CLIP-based OOD detection, guided by a theory of semantic pool size, activation probability, and label dependence.
Problem
Zero-shot OOD detection with a pre-trained VLM selects OOD labels from a semantic pool, but simply enlarging the pool with bigger lexicons fails to consistently improve performance due to uncommon words and near-synonyms.
Approach
Instead of selecting more words from lexicons, construct a conjugated semantic pool of adjective-modified superclass names (e.g., "white creature", "valuable item") that act as cluster centers for samples sharing properties across categories, yielding higher activation probabilities and low mutual dependence with original labels.
Key results
- On ImageNet-1k with CLIP ViT-B/16, the method reports average AUROC 95.76% and FPR95 17.51%, improving over NegLabel by 1.55% AUROC and 7.89% FPR95.
- The CSP alone gives average FPR95 20.67 vs 27.39 for the original pool on the ablation, and the full combination reaches 17.71 FPR95 average.
- Improvements hold across CLIP backbones (ResNet50 to ViT-L/14-336px), with average FPR95 reductions of roughly 5-10 points over NegLabel.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CSP to the original semantic pool | NegLabel with CLIP ViT-B/16 · ImageNet-1k ID, average across iNaturalist, SUN, Places, Textures | FPR95: 25.40 -> 17.51 | Helped |
| Add CSP to the original semantic pool | NegLabel with CLIP ViT-B/16 · ImageNet-1k ID, average across iNaturalist, SUN, Places, Textures | AUROC: 94.21 -> 95.76 | Helped |
| Add CSP to the original semantic pool | NegLabel with CLIP ViT-B/16 · ImageNet-10 ID / ImageNet-100 OOD | FPR95: 2.50 -> 2.22 | Hurt |
| Use CSP instead of simple adjective labels (ablation) | Original semantic pool expansion · ImageNet-1k ID, average across iNaturalist, SUN, Places, Textures | FPR95: 24.36 -> 17.71 | Helped |
| Add CSP to the original semantic pool (ablation) | ImageNet-1k OOD detector · ImageNet-1k ID, average across iNaturalist, SUN, Places, Textures | AUROC: 93.70 -> 95.73 | Helped |
| Add CSP to the original semantic pool (ablation) | ImageNet-1k OOD detector · ImageNet-1k ID, average across iNaturalist, SUN, Places, Textures | FPR95: 27.39 -> 17.71 | Helped |
| Add simple adjective labels to the original semantic pool (ablation) | ImageNet-1k OOD detector · ImageNet-1k ID, average across iNaturalist, SUN, Places, Textures | FPR95: 27.39 -> 24.36 | Helped |
| Use only CSP, removing the original semantic pool (ablation) | ImageNet-1k OOD detector · ImageNet-1k ID, average across iNaturalist, SUN, Places, Textures | FPR95: 17.71 -> 20.67 | Hurt |
| Use only simple adjective labels instead of CSP (ablation) | ImageNet-1k OOD detector · ImageNet-1k ID, average across iNaturalist, SUN, Places, Textures | AUROC: 95.73 -> 94.35 | Hurt |
| Add CSP to the original semantic pool | NegLabel with CLIP ViT-B/16 · iNaturalist OOD | FPR95: 1.91 -> 1.54 | Helped |
| Add CSP to the original semantic pool | NegLabel with CLIP ViT-B/16 · SUN OOD | FPR95: 20.53 -> 13.66 | Helped |
| Add CSP to the original semantic pool | NegLabel with CLIP ViT-B/16 · Places OOD | FPR95: 35.59 -> 29.32 | Helped |
Surprising findings
- Simply enlarging the semantic pool with larger lexicons (Common-20K, Part-of-Speech corpus) does not consistently improve OOD detection performance.
- The CSP slightly hurts performance on iNaturalist, where OOD samples have limited visual diversity.
Limitations
- Effectiveness depends on the assumption that OOD samples exhibit diverse visual properties; gains are small or negative on low-diversity datasets like iNaturalist.
- The theory relies on approximations (Lyapunov CLT with near-independence, linear u(r) assumption, Beta distribution assumptions) that are only partly validated.
Experimental setup
Datasets: ImageNet-1k WordNet iNaturalist SUN Places Textures ImageNet-O WaterBirds Places-bg Stanford-Cars CUB-200 Oxford-Pet
Benchmarks: ImageNet-1k OOD benchmark (iNaturalist, SUN, Places, Textures) ImageNet-O WaterBirds/Placesbg hard OOD tasks Stanford-Cars CUB-200 Oxford-Pet Food-101 ImageNet-Sketch ImageNet-A ImageNet-R ImageNetV2
Metrics: AUROC FPR95
Baselines: MSP ODIN Energy GradNorm ViM KNN VOS ZOC MCM NPOS CoOp CoCoOp
Builds on: NegLabel MCM WordNet CLIP
Closest papers by meaning
- CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection (ICML 2024 · similarity 0.89)
- AdaNeg: Adaptive Negative Proxy Guided OOD Detection with Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Negative Label Guided OOD Detection with Pretrained Vision-Language Models (ICLR 2024 · similarity 0.89)
- Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization (ICLR 2024 · similarity 0.89)
- ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms (ICLR 2024 · similarity 0.89)
- $\Delta \mathrm{Energy}$: Optimizing Energy Change During Vision-Language Alignment Improves both OOD Detection and OOD Generalization (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.