Research map / Vision-language and multimodal / CLIP and zero-shot

VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection

AAAI 2024 · Vision-language and multimodal › CLIP and zero-shot · approach: Fine-grained improvements to CLIP · 206 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adapts a frozen CLIP (ViT-B/16) to weakly supervised video anomaly detection with a dual-branch design (visual classification plus language-image alignment), a local-global temporal adapter, learnable and anomaly-focused visual prompts, and MIL-Align, reaching 84.51% AP on XD-Violence and 88.02% AUC on UCF-Crime.

Problem

Weakly supervised video anomaly detection (WSVAD) requires frame-level anomaly predictions from video-level labels, and existing methods use classification on visual features without exploiting vision-language associations from CLIP.

Approach

Keep CLIP encoders frozen and train only lightweight adapters and prompts, with a dual branch: a visual binary classifier for coarse-grained detection and a text-video alignment branch (MIL-Align) for fine-grained, class-level detection, with an anomaly-focused visual prompt linking the two branches.

Key results

• VadCLIP achieves 84.51% AP on XD-Violence, versus 82.41% for DMU and 82.17% for CLIP-TSA.
• VadCLIP achieves 88.02% AUC on UCF-Crime, above CLIP-TSA (87.58%).
• On fine-grained XD-Violence, VadCLIP reaches 24.70 mAP AVG, versus 20.21 for AVVD.

Measured effects reported in the paper

Surprising findings

• The A-branch alone reaches only 68.15% AP on XD-Violence, far below the C-branch alone (80.53%), and needs the C-branch to reach competitive performance.
• Using fine-grained class labels (AVVD) does not improve coarse-grained performance over binary labels, which the authors attribute to label excess making binary classification harder.

Limitations

• The A-branch alone performs poorly on coarse-grained detection, so the dual-branch design is needed.
• Hyperparameters such as window length and lambda differ across the two datasets (e.g., lambda 1e-4 vs 1e-1).

Experimental setup

Datasets: XD-Violence UCF-Crime

Benchmarks: XD-Violence UCF-Crime

Metrics: AP AUC AnoAUC mAP@IoU AVG mAP

Baselines: SVM baseline OCSVM Hasan et al. Ju et al. (CLIP-based action recognition) Sultani et al. Wu et al. RTFM AVVD DMU CLIP-TSA UMIL

Builds on: CLIP CoOp Multiple instance learning (Sultani et al.) RTFM AVVD

Closest papers by meaning

• Harnessing Large Language Models for Training-free Video Anomaly Detection (CVPR 2024 · similarity 0.92)
• Learning Event Completeness for Weakly Supervised Video Anomaly Detection (ICML 2025 · similarity 0.91)
• VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models (CVPR 2025 · similarity 0.89)
• Federated Weakly Supervised Video Anomaly Detection with Multimodal Prompt (AAAI 2025 · similarity 0.89)
• Open-Vocabulary Video Anomaly Detection (CVPR 2024 · similarity 0.89)
• Ex-VAD: Explainable Fine-grained Video Anomaly Detection Based on Visual-Language Models (ICML 2025 · similarity 0.89)