Research map / Vision-language and multimodal / Cross-modal learning
Beyond Single-Modal Boundary: Cross-Modal Anomaly Detection through Visual Prototype and Harmonization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Transferable Visual Prototypes (global and local normal/abnormal prototypes learned on CLIP visual features per dataset) combined via a Prototype Harmonization weighting for zero-shot cross-modal anomaly detection, plus a Visual Discrepancy Inference for few-shot normal images.
Problem
Anomaly detection for test data from a modality unseen during training; existing CLIP text-prompt zero-shot and few-shot anomaly detectors degrade when auxiliary and test modalities differ.
Approach
Learn normal/abnormal prototypes directly in CLIP's visual embedding space (global and local, per training dataset) rather than in text space, then adaptively weight and aggregate prototypes from multiple modalities by their similarity to the query image; few-shot normal images add a patch-level nearest-neighbor discrepancy score.
Key results
- Zero-shot AUROC gains over best competing method of 4.1% (RGB), 6.1% (3D), 7.6% (MRI/CT) and 6.8% (thermal).
- Few-shot (1,2,4,8 shot) method achieves highest AUROC/AP on all ten datasets; e.g., 2-shot MvTec AD AUROC 96.5 vs 91.7 best baseline (PromptAD 94.8 in 2-shot table, InCTRL 91.1).
- Ablation: global+local prototypes reach 92.3 AUROC on MvTec AD vs 91.6 global-only and 84.6 local-only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine global and local prototypes instead of using the global prototype alone (ablation) | Transferable Visual Prototype · MvTec AD | AUROC: 91.6 -> 92.3 | Helped |
| combine global and local prototypes instead of using the local prototype alone (ablation) | Transferable Visual Prototype · MvTec AD | AUROC: 84.6 -> 92.3 | Helped |
| use the global prototype instead of both global and local prototypes (ablation) | Transferable Visual Prototype · OCT2017 | AUROC: 74.2 -> 91.2 | Hurt |
| use the local prototype instead of both global and local prototypes (ablation) | Transferable Visual Prototype · OCT2017 | AUROC: 96.6 -> 91.2 | Helped |
| use prototype harmonization rather than the best single prototype (ablation) | zero-shot inference · MvTec AD | AUROC: 90.9 -> 92.3 | Helped |
| use prototype harmonization rather than maximum-relevance prototype selection (ablation) | zero-shot inference · MvTec AD | AUROC: 89.9 -> 92.3 | Helped |
| use prototype harmonization rather than equal-weight averaging (ablation) | zero-shot inference · MvTec AD | AUROC: 90.8 -> 92.3 | Helped |
| train TVPs per dataset instead of one TVP per modality (ablation) | Transferable Visual Prototype · MvTec AD | AUROC: 90.8 -> 92.3 | Helped |
| train TVPs on sub-datasets instead of whole datasets (ablation) | Transferable Visual Prototype · MvTec AD | AUROC: 92.0 -> 92.3 | Helped |
| add Visual Discrepancy Inference to TVP in the two-shot setting (ablation) | Ours · MvTec AD | AUROC: 92.3 -> 96.5 | Helped |
| add TVP to Visual Discrepancy Inference in the two-shot setting (ablation) | Ours · MvTec AD | AUROC: 95.7 -> 96.5 | Helped |
| use the proposed cross-modal method rather than the strongest listed baseline | zero-shot anomaly detection · MvTec AD | AUROC: 91.8 -> 92.3 | Helped |
Surprising findings
- Local-only prototypes outperform global-only on OCT2017 (96.6 vs 92.0 AUROC in ablation table).
Limitations
- Zero-shot performance is weaker on some datasets (e.g., Leaves vs. AnomalyCLIP-like methods on certain tasks is not uniformly best; WinCLIP is stronger on MvTec AD AP in some settings).
- Few-shot setting requires a few normal images from the target modality at test time.
- Evaluated on only ten datasets across four modalities with a single CLIP backbone and a fixed 224x224 resolution.
Experimental setup
Datasets: MvTec AD VisA BTAD MPDD MvTec 3D-AD Eyecandies Brain MRI OCT2017 WTB Leaves
Benchmarks: MvTec AD VisA BTAD MPDD MvTec 3D-AD Eyecandies Brain MRI OCT2017 WTB Leaves
Metrics: AUROC AP
Baselines: VAND WinCLIP AnomalyCLIP AdaCLIP PatchCore RegAD PromptAD InCTRL
Builds on: CLIP WinCLIP AnomalyCLIP PatchCore
Closest papers by meaning
- Rethinking Reverse Distillation for Multi-Modal Anomaly Detection (AAAI 2024 · similarity 0.88)
- Align3D-AD: Cross-Modal Feature Alignment and Dual-Prompt Learning for Zero-shot 3D Anomaly Detection (NeurIPS 2026 · similarity 0.87)
- Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.86)
- UniVAD: A Training-free Unified Model for Few-shot Visual Anomaly Detection (CVPR 2025 · similarity 0.86)
- CNC: Cross-modal Normality Constraint for Unsupervised Multi-class Anomaly Detection (AAAI 2025 · similarity 0.86)
- Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.