Research map / Vision-language and multimodal / CLIP and zero-shot
Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts CLIP (ViT-L/14) for zero-/few-shot medical anomaly classification and segmentation by inserting multi-level residual adapters trained with pixel-wise visual-language alignment losses, combined with zero-shot text comparison and few-shot memory-bank comparison.
Problem
Zero- and few-shot anomaly detection in medical images fails to transfer from natural-image vision-language models because of large domain shift between natural and medical modalities and anatomical regions.
Approach
Lightweight residual bottleneck adapters are inserted at multiple stages of a frozen CLIP visual encoder and trained with multi-level pixel-wise and image-level visual-language alignment losses against normal/abnormal text prompts, so the backbone is adapted rather than just projected; at test time zero-shot text comparison is fused with few-shot memory-bank nearest-neighbour scores.
Key results
- Zero-shot: average AUC gain of 6.24% (AC) and 2.03% (AS) over April-GAN across medical datasets.
- Few-shot (K=4): average AUC gain of 7.33% (AC) and 2.37% (AS) over April-GAN.
- Zero-shot AC on HIS/OCT17/RESC reaches 77.90/95.40/83.31 AUC versus April-GAN's 72.36/92.61/75.67.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adapt CLIP with multi-level feature adapters rather than independently optimized multi-level projectors (ablation) | CLIP ViT-L/14 · Medical zero-shot benchmarks | AC AUC: +13.57% average improvement | Helped |
| adapt CLIP with multi-level feature adapters rather than independently optimized multi-level projectors (ablation) | CLIP ViT-L/14 · BrainMRI, LiverCT, RESC | AS AUC: 0.88%, 0.21%, and 2.88% | Helped |
| combine features from all layers instead of using an individual layer (ablation) | MVFA · Medical few-shot (K=4) | AC AUC: 88.97 vs 88.84 | Helped |
| combine features from all layers instead of using an individual layer (ablation) | MVFA · Medical few-shot (K=4) | AS AUC: 98.67 vs 98.62 | Helped |
| train with multi-level rather than single-layer adapters (ablation) | MVFA · Medical few-shot (K=4) | AC and AS AUC | Helped |
| use multi-level adapters for zero-shot medical anomaly detection instead of APRIL-GAN | CLIP-based medical AD · Medical benchmarks | AC AUC: 6.24% average improvement | Helped |
| use multi-level adapters for zero-shot medical anomaly detection instead of APRIL-GAN | CLIP-based medical AD · Medical benchmarks | AS AUC: 2.03% average improvement | Helped |
| use multi-level adapters for few-shot medical anomaly detection instead of APRIL-GAN | CLIP-based medical AD · Medical benchmarks | AC AUC: 7.33% average improvement | Helped |
| use multi-level adapters for few-shot medical anomaly detection instead of APRIL-GAN | CLIP-based medical AD · Medical benchmarks | AS AUC: 2.37% average improvement | Helped |
Surprising findings
- MedCLIP outperforms the proposed method on ChestXray, attributed to its training on overlapping ChestXray data.
- Feature projectors (alignment only, without backbone adaptation) perform far worse than adapters in zero-shot AC, e.g. OCT17 AC 49.85 vs 95.40.
Limitations
- Focuses on medical AD; in-domain industrial performance is only comparable, not superior.
- Relies on a few annotated abnormal samples in the few-shot setting.
- Zero-shot performance on ChestXray is weaker than MedCLIP, which saw overlapping data.
Experimental setup
Datasets: BrainMRI LiverCT RESC OCT17 ChestXray HIS BMAD MVTec AD
Benchmarks: BrainMRI LiverCT RESC OCT17 ChestXray HIS MVTec AD
Metrics: AUC
Baselines: CFlowAD RD4AD PatchCore MKD CLIP MedCLIP WinCLIP DRA BGAD APRIL-GAN
Builds on: CLIP WinCLIP APRIL-GAN AdaptFormer-style residual adapters
Closest papers by meaning
- Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Prompts (CVPR 2024 · similarity 0.89)
- Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.88)
- MetaUAS: Universal Anomaly Segmentation with One-Prompt Meta-Learning (NeurIPS 2024 · similarity 0.87)
- Decomposing Disease Descriptions for Enhanced Pathology Detection: A Multi-Aspect Vision-Language Pre-training Framework (CVPR 2024 · similarity 0.87)
- UniVAD: A Training-free Unified Model for Few-shot Visual Anomaly Detection (CVPR 2025 · similarity 0.86)
- Language-Assisted Feature Transformation for Anomaly Detection (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.