Research map / Video / Video segmentation and tracking
Open-Vocabulary Video Anomaly Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces open-vocabulary video anomaly detection (OVVAD), decoupling detection and category classification on frozen CLIP features, with LLM semantic knowledge injection and LLM/AIGC-based pseudo novel anomaly synthesis.
Problem
Weakly supervised video anomaly detection is limited to closed anomaly categories; open-set VAD detects unseen anomalies but cannot name their categories. The paper targets detecting and categorizing both seen and unseen anomalies under weak supervision.
Approach
Decouples OVVAD into class-agnostic detection and class-specific CLIP-based categorization, adds a nearly weight-free temporal adapter and an LLM-driven semantic knowledge injection module for detection, and synthesizes pseudo unseen anomaly videos from LLM text and generative image/video models to fine-tune categorization and detection.
Key results
- Achieves 86.40% AUC on UCF-Crime, 66.53% AP on XD-Violence, and 62.94% AUC on UBnormal, outperforming the closest weakly supervised baselines (e.g., DMU 85.14% AUC, 63.90% AP).
- Ablation on XD-Violence shows the temporal adapter improves AP by 14.47% for novel categories.
- Cross-dataset test: training on UCF-Crime and testing on XD-Violence gives 82.86% AP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the temporal adapter (TA) (ablation) | OVVAD model · UCF-Crime | AUC: 84.79 -> 85.14 | Helped |
| add semantic knowledge injection (SKI) (ablation) | OVVAD model · UCF-Crime | AUC: 84.79 -> 85.04 | Helped |
| add anomaly synthesis (NAS) to TA and SKI (ablation) | OVVAD model · UCF-Crime | AUC: 85.81 -> 86.40 | Helped |
| add the temporal adapter (TA) (ablation) | OVVAD model · XD-Violence | AP: 53.11 -> 60.13 | Helped |
| add semantic knowledge injection (SKI) (ablation) | OVVAD model · XD-Violence | AP: 53.11 -> 56.62 | Helped |
| add anomaly synthesis (NAS) to TA and SKI (ablation) | OVVAD model · XD-Violence | AP: 65.60 -> 66.53 | Helped |
| add the temporal adapter (TA) (ablation) | OVVAD model · UBnormal | AUC: 60.51 -> 61.18 | Helped |
| add semantic knowledge injection (SKI) (ablation) | OVVAD model · UBnormal | AUC: 60.51 -> 61.93 | Helped |
| add anomaly synthesis (NAS) to TA and SKI (ablation) | OVVAD model · UBnormal | AUC: 61.67 -> 62.94 | Helped |
| fine-tune using novel pseudo anomalies plus base anomalies rather than novel pseudo anomalies only (ablation) | OVVAD categorization · UCF-Crime | TOP1 accuracy: 39.29 -> 41.43 | Helped |
| fine-tune using novel pseudo anomalies plus base anomalies rather than novel pseudo anomalies only (ablation) | OVVAD categorization · XD-Violence | TOP1 accuracy: 62.03 -> 64.68 | Helped |
| use the proposed OVVAD model instead of DMU | video anomaly detection · UCF-Crime | AUC: 85.14 -> 86.40 | Helped |
Surprising findings
- Using only generated novel samples during fine-tuning degrades base-category performance, so a reasonable fine-tuning scheme combining base and pseudo data is needed.
- The nearly weight-free temporal adapter (a graph-convolution-style, distance-based module) outperforms transformer-style temporal modules on novel categories.
Limitations
- Categorization accuracy remains low on some categories, especially on UCF-Crime per the confusion matrices.
- Only single-category videos are supported; multi-category XD-Violence videos were excluded.
- Generated pseudo anomalies are images animated into clips or short generated videos rather than high-fidelity videos.
Experimental setup
Datasets: UCF-Crime XD-Violence UBnormal
Benchmarks: UCF-Crime XD-Violence UBnormal
Metrics: AUC AP top-1 accuracy
Baselines: Sultani et al. (MIL) Wu et al. (XD-Violence) RTFM DMU Zhu et al. (open-set VAD) Georgescu et al. OCSVM UMIL CLIP-TSA
Builds on: CLIP CoOp multiple instance learning Top-K mechanism
Closest papers by meaning
- Harnessing Large Language Models for Training-free Video Anomaly Detection (CVPR 2024 · similarity 0.92)
- MoniTor: Exploiting Large Language Models with Instruction for Online Video Anomaly Detection (NeurIPS 2025 · similarity 0.90)
- Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity (CVPR 2025 · similarity 0.90)
- Advancing Video Anomaly Detection: A Concise Review and a New Dataset (NeurIPS 2024 · similarity 0.90)
- Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thought (NeurIPS 2025 · similarity 0.89)
- HAWK: Learning to Understand Open-World Video Anomalies (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.