Research map / Vision-language and multimodal / Multimodal LLMs

Towards Zero-Shot Anomaly Detection and Reasoning with Multimodal Large Language Models

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Benchmarks and analysis of MLLMs · 38 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds Anomaly-Instruct-125k instruction data and the VisA-D&R benchmark, and proposes Anomaly-OV, a LLaVA-OneVision-based MLLM with a look-twice feature matching anomaly expert that selects suspicious visual tokens for zero-shot anomaly detection and reasoning.

Problem

Zero-shot anomaly detection is usually done with CLIP-style scoring that gives no rationale, and generalist MLLMs such as GPT-4o miss fine-grained anomalies and cannot explain them, partly due to lack of anomaly instruction data and benchmarks.

Approach

An anomaly expert built on the MLLM's own visual encoder (no text encoder) computes patch-level anomaly significance maps via look-back global features and learnable positive/negative embeddings; the LLM then receives emphasized suspicious visual tokens with an adverb indication prompt.

Key results

• Anomaly expert achieves 88.6 average image-level AUROC across nine ZSAD benchmarks, versus 85.3 for the best baseline AdaCLIP.
• Anomaly-OV-7B reaches 0.79 accuracy and 0.83 F1 on VisA-D&R anomaly detection, the best among listed MLLMs, with GPT-4o at 0.70 accuracy and 0.68 F1.
• Anomaly-OV-0.5B improves accuracy by 0.08 and F1 by 0.06 over the fine-tuned LLaVA-OV-0.5B base, per the paper's text.

Measured effects reported in the paper

Surprising findings

• GPT-4o fails to detect many anomalies, but given a bounding-box or textual hint it describes them well, suggesting the anomaly information is in the visual tokens and the LLM fails to select them.
• The anomaly expert uses only the visual encoder with no text encoder, and the paper credits the extra WebAD in-the-wild data more than the architecture for ZSAD gains.

Limitations

• Detection performance is limited by the anomaly expert, with recall and accuracy still far from ideal.
• Existing industrial anomaly datasets have limited anomaly type diversity, causing fine-grained description failures and hallucinations.
• VisA-D&R covers only industrial images, so medical and 3D evaluation is qualitative or limited.

Experimental setup

Datasets: Anomaly-Instruct-125k MVTec AD VisA BMAD Anomaly-ShapeNet Real3D-AD MVTec-3D AD AITEX ELPV BTAD MPDD BrainMRI

Benchmarks: MVTec AD VisA AITEX ELPV BTAD MPDD BrainMRI HeadCT Br35H VisA-D&R

Metrics: AUROC Accuracy Precision Recall F1-score ROUGE-L SBERT similarity GPT-Score

Baselines: CLIP CoOp WinCLIP APRIL-GAN AnoVL AnomalyCLIP AdaCLIP GPT-4V GPT-4o Qwen2-VL-2B Qwen2-VL-7B InternVL-2-8B

Builds on: LLaVA-OneVision AnomalyCLIP Q-Former LLaVA

Closest papers by meaning

• AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection (ICLR 2024 · similarity 0.91)
• LogicAD: Explainable Anomaly Detection via VLM-based Text Feature Extraction (AAAI 2025 · similarity 0.89)
• DZAD: Diffusion-based Zero-shot Anomaly Detection (AAAI 2025 · similarity 0.89)
• Aligning and Prompting Anything for Zero-Shot Generalized Anomaly Detection (AAAI 2025 · similarity 0.88)
• AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models (AAAI 2024 · similarity 0.88)
• Unseen Visual Anomaly Generation (CVPR 2025 · similarity 0.88)