Research map / Detection and segmentation / Open-set and few-shot detection
AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts CLIP for zero-shot anomaly detection by learning two object-agnostic learnable text prompts (normal/abnormal) with glocal image- and pixel-level losses, plus a diagonally prominent attention modification for local features.
Problem
Zero-shot anomaly detection and segmentation across domains with very different foreground objects, where CLIP's object-centric alignment yields weak anomaly recognition.
Approach
Learn object-agnostic normality and abnormality text prompts (class name replaced by the word "object") using global and local (focal + Dice) losses on auxiliary data, with a frozen CLIP and a Diagonally Prominent Attention Map (DPAM) that replaces Q-K attention with V-V attention for sharper local features.
Key results
- On industrial datasets AnomalyCLIP reports image-level AUROC 91.5 on MVTec AD and 82.1 on VisA, and pixel-level AUROC 91.1 on MVTec AD, outperforming WinCLIP and VAND in most datasets.
- On 10 medical datasets trained only on MVTec AD, AnomalyCLIP achieves best results on all, e.g., HeadCT image AUROC 93.4 and TN3K pixel AUROC 81.5.
- AnomalyCLIP with V-V DPAM, frozen CLIP, and 12 learnable word embeddings is the default configuration; object-agnostic prompts beat object-aware prompts on most datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use object-agnostic rather than object-aware text prompts (ablation) | AnomalyCLIP · ZSAD image- and pixel-level datasets | AUROC | Helped |
| remove DPAM (ablation) | AnomalyCLIP · Anomaly segmentation | segmentation performance | Hurt |
| remove object-agnostic text prompts (ablation) | AnomalyCLIP · diverse unseen objects | anomaly detection performance | Hurt |
| remove learnable tokens in text encoder (ablation) | AnomalyCLIP · ZSAD | overall performance | Hurt |
| remove global context optimization (ablation) | AnomalyCLIP · image-level anomaly detection | detection performance | Hurt |
| remove local context optimization (ablation) | AnomalyCLIP · pixel-level anomaly detection | segmentation performance | Hurt |
| use global and local context optimization together instead of either alone (ablation) | AnomalyCLIP · ZSAD | overall performance | Helped |
| replace V-V attention with Q-Q attention (ablation) | AnomalyCLIP · image-level anomaly detection | detection performance | Hurt |
| replace V-V attention with K-K attention (ablation) | AnomalyCLIP · pixel-level anomaly detection | segmentation performance | Hurt |
| increase prompt length beyond 12 to 16 (ablation) | AnomalyCLIP · ZSAD | detection and segmentation performance | Hurt |
| replace 'damaged' with similar anomaly words (ablation) | AnomalyCLIP · MVTec AD, VisA, MPDD, BTAD, SDD, DAGM, DTD-Synthetic and medical datasets | AUROC: 91.5 -> 91.4 (MVTec AD image-level); 82.1 -> 80.7 (VisA image-level); 93.4 -> 93.1 (HeadCT image-level) | No clear effect |
| replace 'object' with the target class name (ablation) | AnomalyCLIP · ZSAD | performance | Hurt |
Surprising findings
- Medical ZSAD performance is comparatively low when the prompt is tuned on industrial MVTec AD data, which the authors attribute to domain mismatch of auxiliary data.
- Replacing the word 'damaged' with other anomaly words (anomalous, flawed, defective, blemished) changes results only slightly, indicating insensitivity to the prompt token choice.
Limitations
- Relies on auxiliary AD data (e.g., MVTec AD or VisA test data) for prompt tuning, so it is not strictly zero-shot with respect to all target data.
- Weaker performance on medical datasets when tuned on industrial data; the authors note the domain gap.
- Code is promised upon acceptance only; the paper does not release it at submission.
Experimental setup
Datasets: MVTec AD VisA MPDD BTAD SDD DAGM DTD-Synthetic ISIC CVC-ClinicDB CVC-ColonDB Kvasir Endo
Benchmarks: MVTec AD VisA MPDD BTAD SDD DAGM DTD-Synthetic ISIC CVC-ClinicDB CVC-ColonDB Kvasir Endo
Metrics: AUROC AP AUPRO
Baselines: CLIP CLIP-AC WinCLIP VAND CoOp PatchCore RD4AD
Builds on: CLIP CoOp WinCLIP VAND
Closest papers by meaning
- Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.91)
- Aligning and Prompting Anything for Zero-Shot Generalized Anomaly Detection (AAAI 2025 · similarity 0.90)
- AA-CLIP: Enhancing Zero-Shot Anomaly Detection via Anomaly-Aware CLIP (CVPR 2025 · similarity 0.89)
- DZAD: Diffusion-based Zero-shot Anomaly Detection (AAAI 2025 · similarity 0.87)
- Bayesian Prompt Flow Learning for Zero-Shot Anomaly Detection (CVPR 2025 · similarity 0.87)
- Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Prompts (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.