Research map / Vision-language and multimodal / CLIP and zero-shot
AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AnomalyGPT, which fine-tunes a PandaGPT-initialized vision-language model with prompt embeddings and a feature-matching decoder on simulated anomaly data to detect and localize industrial anomalies without manual thresholds.
Problem
Industrial anomaly detection methods output only anomaly scores that need per-class manual thresholds, and general LVLMs lack domain knowledge and fine-grained localization for detecting defects.
Approach
Combine a frozen ImageBind image encoder and Vicuna-7B LLM with a lightweight patch-text feature-matching decoder whose localization maps are converted into prompt embeddings by a learnable prompt learner, trained with simulated (Poisson-edited cut-paste) anomalies and alternated with PandaGPT pre-training data so the LLM directly judges anomaly presence and grid location.
Key results
- With one normal shot on MVTec-AD after VisA training, AnomalyGPT reaches 86.1% accuracy, 94.1% image AUC and 95.3% pixel AUC.
- Unsupervised on MVTec-AD, AnomalyGPT reports 97.4% image AUC, 93.1% pixel AUC and 93.3% accuracy, versus UniAD's 96.5% image AUC.
- On VisA one-shot, AnomalyGPT achieves 87.4% image AUC and 77.4% accuracy, with WinCLIP leading pixel AUC at 96.4% versus AnomalyGPT's 96.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the lightweight feature-matching decoder (ablation) | AnomalyGPT · MVTec-AD (unsupervised) | Pixel-AUC: from ~72.2 accuracy-only baseline to 90.9 pixel-AUC with decoder | Helped |
| replace LLM output with max-normal-score threshold (remove LLM) (ablation) | AnomalyGPT decoder · MVTec-AD (unsupervised) | Accuracy: 90.3 without LLM vs 93.3 full | Hurt |
| use prompt tuning (prompt learner) instead of LoRA for LLM fine-tuning (ablation) | AnomalyGPT · VisA (1-shot) | Accuracy: 72.7 with prompt learner+LoRA vs 64.7 decoder+LoRA; 77.4 full (prompt tuning, no LoRA) | Helped |
| add LoRA fine-tuning of LLM (ablation) | AnomalyGPT without decoder · MVTec-AD (unsupervised) | Accuracy: 72.2 -> 79.8 accuracy | Helped |
| add prompt learner to decoder+LLM with LoRA (ablation) | AnomalyGPT · MVTec-AD (unsupervised) | Accuracy: 84.2 -> 83.9 accuracy; image-AUC 97.1 -> 96.0 | Hurt |
| add prompt learner with LLM (no LoRA) and decoder (ablation) | AnomalyGPT · VisA (1-shot) | Accuracy: 56.5 -> 56.6 without decoder; 75.4 -> 77.4 with decoder configuration in full model | Helped |
| AnomalyGPT vs WinCLIP one-shot transfer | AnomalyGPT · MVTec-AD 1-shot (trained on VisA) | Accuracy: 86.1 accuracy; WinCLIP reports no accuracy (threshold-based scores only) | Helped |
| AnomalyGPT vs PatchCore/WinCLIP few-shot image-level AUC | AnomalyGPT · MVTec-AD 1-shot | Image-AUC: 94.1 vs WinCLIP 93.1 and PatchCore 83.4 | Helped |
| AnomalyGPT vs WinCLIP few-shot pixel-AUC | AnomalyGPT · VisA 1-shot | Pixel-AUC: 96.2 vs WinCLIP 96.4 | No clear effect |
Surprising findings
- PatchCore reaches 99.3% image AUC on MVTec-AD but only 79.76% accuracy under a unified threshold, suggesting threshold selection is a major practical bottleneck.
- Prompt tuning outperformed LoRA fine-tuning of the LLM in accuracy and transferability in the ablation.
Limitations
- Few-shot localization is slightly lower than unsupervised localization because no parameters are trained in the one-shot setting.
- Unified-setting comparisons rely on baseline numbers reported from prior papers rather than re-run experiments.
Experimental setup
Datasets: MVTec-AD VisA PandaGPT pre-training data NSA-simulated anomaly images
Benchmarks: MVTec-AD VisA
Metrics: image-level AUC pixel-level AUC accuracy
Baselines: SPADE PaDiM PatchCore WinCLIP UniAD JNLD MiniGPT-4 LLaVA PandaGPT
Builds on: PandaGPT WinCLIP PatchCore NSA LLaVA
Closest papers by meaning
- MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection (ICLR 2025 · similarity 0.91)
- LogicAD: Explainable Anomaly Detection via VLM-based Text Feature Extraction (AAAI 2025 · similarity 0.89)
- Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection (CVPR 2024 · similarity 0.89)
- AnomalyNCD: Towards Novel Anomaly Class Discovery in Industrial Scenarios (CVPR 2025 · similarity 0.89)
- Toward Generalist Anomaly Detection via In-context Residual Learning with Few-shot Sample Prompts (CVPR 2024 · similarity 0.89)
- Towards Training-free Anomaly Detection with Vision and Language Foundation Models (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.