Research map / Detection and segmentation / Open-set and few-shot detection
MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MMAD, a multiple-choice VQA benchmark (39,672 questions over 8,366 industrial images, 7 subtasks) to evaluate MLLMs on industrial anomaly detection, and tests training-free boosts (RAG and expert-model visual prompts).
Problem
MLLMs' ability to act as industrial quality inspectors has not been systematically evaluated, and existing IAD datasets lack the semantic annotations needed to test MLLMs directly.
Approach
A GPT-4V-driven pipeline that adds visual (red mask) and textual prompts plus a retrieved normal template to generate captions and multiple-choice questions over public IAD images, followed by manual filtering; then a broad MLLM evaluation across seven anomaly and object subtasks with 0/1/1+-shot settings.
Key results
- GPT-4o reached the highest average accuracy of 74.9% on MMAD, while the best open-source model InternVL2-76B reached 70.8%.
- Human experts scored 86.65% average and ordinary participants 78.69% on a 177-example subset, above the best MLLMs in anomaly-related tasks.
- RAG with domain knowledge improved InternVL2-76B average accuracy from 70.75% to 76.18%, with the largest gains in defect classification and object classification.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RAG domain knowledge (ablation) | InternVL2-40B · MMAD | average accuracy: 69.59 -> 76.35 (+6.75) | Helped |
| add RAG domain knowledge (ablation) | InternVL2-76B · MMAD | average accuracy: 70.75 -> 76.18 (+5.43) | Helped |
| add RAG domain knowledge (ablation) | LLaVA-NeXT-34B · MMAD | Defect Classification accuracy: 48.79 -> 68.22 (+19.43) | Helped |
| use a retrieved normal image as the template instead of no template (ablation) | InternVL2-76B · MMAD | average accuracy: 69.41 -> 71.31 (+1.90) | Helped |
| use a retrieved normal image as the template instead of no template (ablation) | LLaVA-NeXT-34B · MMAD | average accuracy: 68.45 -> 67.16 (-1.28) | Hurt |
| add an abnormal image in-context (ablation) | InternVL2-40B · MMAD | Anomaly Discrimination accuracy: 63.84 -> 66.47 | Helped |
| increase normal examples from 1-shot to 2-shot (ablation) | InternVL2-40B · MMAD | average accuracy: 69.67 -> 69.66 | No clear effect |
| add chain-of-thought reasoning (ablation) | InternVL2-40B · MMAD | average accuracy: 69.59 -> 65.82 | Hurt |
| add chain-of-thought reasoning (ablation) | InternVL2-76B · MMAD | average accuracy: 70.75 -> 72.29 | Helped |
| disable visual input (ablation) | InternVL2-40B · MMAD | average accuracy: 69.67 -> 45.41 | Hurt |
| disable visual input (ablation) | InternVL2-76B · MMAD | average accuracy: 70.75 -> 44.59 | Hurt |
| use AnomalyCLIP or PatchCore visual expert outputs (ablation) | InternVL2-40B · MMAD | Anomaly Discrimination, Defect Classification, and Defect Description accuracy | Mixed |
Surprising findings
- Most MLLMs fail to exploit the normal template image: 1-shot+ often lowers performance relative to 0-shot, and AnomalyGPT, trained specifically for IAD, performs worst overall.
- Chain-of-thought prompting did not improve anomaly discrimination for InternVL2-76B and degraded InternVL2-40B.
Limitations
- Human-check of generated questions covered only a 177-example subset for the human baseline evaluation, a preliminary study.
- Domain knowledge and generated captions rely on GPT-4V, which may struggle with industrial images and can introduce errors that manual filtering must correct.
Experimental setup
Datasets: MVTec AD MVTec LOCO AD VisA GoodsAD Defect Spectrum
Benchmarks: MMAD
Metrics: accuracy recall precision F1
Baselines: Random Chance Human (expert) Human (ordinary) AnomalyGPT Qwen-VL-Chat LLaVA-1.5 Cambrian-1 SPHINX LLaVA-NeXT LLaVA-NEXT-Interleave InternLM-XComposer2-VL LLaVA-OneVision
Builds on: MVTec AD VisA MVTec LOCO AD GoodsAD Defect Spectrum
Closest papers by meaning
- AnomalyGPT: Detecting Industrial Anomalies Using Large Vision-Language Models (AAAI 2024 · similarity 0.91)
- Towards Open-Vocabulary Industrial Defect Understanding with a Large-Scale Multimodal Dataset (CVPR 2026 · similarity 0.89)
- Zero-Shot Learning in Industrial Scenarios: New Large-Scale Benchmark, Challenges and Baseline (AAAI 2025 · similarity 0.88)
- LogicAD: Explainable Anomaly Detection via VLM-based Text Feature Extraction (AAAI 2025 · similarity 0.88)
- Towards Training-free Anomaly Detection with Vision and Language Foundation Models (CVPR 2025 · similarity 0.88)
- AD-FM: Multimodal LLMs for Anomaly Detection via Multi-Stage Reasoning and Fine-Grained Reward Optimization (AAAI 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.