Research map / Label-efficient and robust learning / OOD and anomaly detection
Overcoming Shortcut Problem in VLM for Robust Out-of-Distribution Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OSPCoOp, a few-shot CLIP prompt-learning method that decouples foreground/background with segmentation and inpainting to create pseudo-OOD background samples and uses masks to regularize ID-irrelevant regions, and introduces the ImageNet-Bg background-OOD benchmark.
Problem
CLIP-based OOD detection suffers from a shortcut problem where foreground and background semantics are coupled, so background-only OOD images are misclassified as in-distribution, and existing few-shot prompt-tuning methods fail to address it.
Approach
Decouple images into ID-relevant foreground and ID-irrelevant background via segmentation and inpainting to synthesize pseudo-OOD background samples and background-replaced ID samples, then use the decoupling masks to apply entropy-based regularization to ID-irrelevant patches, rather than CLIP-similarity-based region selection as in LoCoOp.
Key results
- On ImageNet-1K with ViT-B/16 in 16-shot, OSPCoOp reaches 94.75% average AUROC and 25.13% FPR95, versus 93.52% AUROC for LoCoOp.
- On the proposed ImageNet-Bg benchmark, OSPCoOp achieves mean AUROC 91.30% and FPR95 33.84%, outperforming LoCoOp (about 89.9-90.0 AUROC).
- In the 1-shot setting, OSPCoOp still achieves mean AUROC 93.81% and FPR95 27.73% on ImageNet-1K OOD datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OSPCoOp instead of LAPT | CLIP ViT-B/16 OOD detection · SUN | AUROC: 96.01 -> 96.74 | Helped |
| Use OSPCoOp instead of EOE | CLIP ViT-B/16 OOD detection · Places | AUROC: 92.94 -> 94.01 | Helped |
| Use OSPCoOp instead of DPM-F | CLIP ViT-B/16 OOD detection · ImageNet-Bg | AUROC: 89.53 -> 91.37 | Helped |
| Add decoupled background OOD augmentation (ablation) | OSPCoOp · Four traditional OOD datasets (Avg) | AUROC: 92.97 -> 94.65 | Helped |
| Add local-repeat OOD augmentation (ablation) | OSPCoOp · Four traditional OOD datasets (Avg) | AUROC: 92.97 -> 93.43 | Helped |
| Combine background and local-repeat OOD augmentation (ablation) | OSPCoOp · Four traditional OOD datasets (Avg) | AUROC: 94.65 -> 94.75 | Helped |
| Add ID augmentation (ablation) | OSPCoOp · Four traditional OOD datasets (Avg), one-shot | AUROC: 93.81 -> 93.92 | Helped |
| Add ID augmentation (ablation) | OSPCoOp · Four traditional OOD datasets (Avg), two-shot | AUROC: 93.51 -> 93.91 | Helped |
| Add ID augmentation (ablation) | OSPCoOp · Four traditional OOD datasets (Avg), four-shot | AUROC: 93.86 -> 94.16 | Helped |
| Add OOD-region irrelevance loss (ablation) | OSPCoOp · ImageNet-1K OOD (Avg) | AUROC: 92.05 -> 93.94 | Helped |
| Add OOD-global irrelevance loss (ablation) | OSPCoOp · ImageNet-1K OOD (Avg) | AUROC: 92.05 -> 94.14 | Helped |
| Use both OOD-region and OOD-global irrelevance losses (ablation) | OSPCoOp · ImageNet-1K OOD (Avg) | AUROC: 92.05 -> 94.75 | Helped |
Surprising findings
- Methods such as CLIPN and outlier-label exposure (NegLabel, LAPT, OLE) perform worse on ImageNet-Bg than zero-shot MCM, and DPM-T drops markedly relative to DPM-F, suggesting few-shot parameter insertion can worsen shortcuts.
- Using CLIP feature similarity to delineate ID-irrelevant regions (LoCoOp-style rank) slightly hurts performance when combined with the mask method, indicating it reinforces background shortcuts.
Limitations
- ImageNet-Bg remains limited in scale
- Theoretical analysis of the shortcut problem is lacking
- Filtering of inpainted backgrounds relies on CLIP similarity, which may retain some ID content
Experimental setup
Datasets: ImageNet-1K iNaturalist SUN Places Texture ImageNet-Bg
Benchmarks: ImageNet-1K OOD (iNaturalist, SUN, Places, Texture) ImageNet-Bg ImageNet-Bg(S)
Metrics: AUROC FPR95
Baselines: ZOC MCM GL-MCM CLIPN-A DPM-F NegLabel LAPT EOE OLE CoOp LoCoOp SCT
Builds on: CoOp LoCoOp GL-MCM
Closest papers by meaning
- Image Background Serves as Good Proxy for Out-of-distribution Data (ICLR 2024 · similarity 0.88)
- RUNA: Object-Level Out-of-Distribution Detection via Regional Uncertainty Alignment of Multimodal Representations (AAAI 2025 · similarity 0.88)
- A Systematic Analysis of Out-of-Distribution Detection Under Representation and Training Paradigm Shifts (NeurIPS 2026 · similarity 0.87)
- Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detection (ICLR 2025 · similarity 0.87)
- ID-like Prompt Learning for Few-Shot Out-of-Distribution Detection (CVPR 2024 · similarity 0.87)
- Enhanced OoD Detection through Cross-Modal Alignment of Multi-Modal Representations (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.