Research map / Video / Video segmentation and tracking
Just Dance with pi! A Poly-modal Inductor for Weakly-supervised Video Anomaly Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes π-VAD, a weakly-supervised video anomaly detector that uses five auxiliary modalities (pose, depth, panoptic masks, optical flow, text) only during training via pseudo-modality generation and cross-modal induction, keeping RGB-only inference.
Problem
Weakly-supervised video anomaly detection relies on RGB spatio-temporal features, which are not distinctive enough for complex human-centered anomalies such as shoplifting, and multi-modal foundation models are data-hungry and add inference cost.
Approach
A teacher-student architecture where a Poly-modal Inductor generates pseudo-modality embeddings directly from RGB features (PMG), aligns them to RGB via bi-directional InfoNCE, and distills task-aware multi-modal features from a frozen UR-DMU teacher (CMI), so extra modality backbones are needed only at training time.
Key results
- On UCF-Crime, π-VAD reaches 90.33% AUC, versus 86.97% for the UR-DMU baseline and 88.02% for multi-modal VadCLIP.
- On XD-Violence, π-VAD reaches 85.37% AP, outperforming VadCLIP's 84.15%.
- On MSAD, π-VAD reaches 88.68% AUC versus 85.78% for the authors' UR-DMU reimplementation, and runs at 30.51 FPS at inference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add five auxiliary modalities through PI | UR-DMU · UCF-Crime | AUC: +2.75% | Helped |
| Add five auxiliary modalities through PI | UR-DMU · XD-Violence | AP: +1.22% | Helped |
| Add five auxiliary modalities through PI | UR-DMU · MSAD | AUC: +1.65% | Helped |
| Add PMG, alignment, and distillation | UR-DMU · UCF-Crime | AUC: 86.97 -> 90.33 | Helped |
| Remove CMI alignment and distillation (ablation) | π-VAD · UCF-Crime | AUC: 90.33 -> 84.66 | Hurt |
| Remove distillation while retaining alignment (ablation) | π-VAD · UCF-Crime | AUC: 90.33 -> 85.84 | Hurt |
| Remove alignment while retaining distillation (ablation) | π-VAD · UCF-Crime | AUC: 90.33 -> 86.29 | Hurt |
| Remove PMG (ablation) | π-VAD · UCF-Crime | AUC: 90.33 -> 90.58 | No clear effect |
| Use PI at early stage only (ablation) | UR-DMU · UCF-Crime | AUC: 90.33 -> 87.14 | Hurt |
| Use PI at late stage only (ablation) | UR-DMU · UCF-Crime | AUC: 90.33 -> 87.48 | Hurt |
| Add pose modality (ablation) | UR-DMU · UCF-Crime | AUC: 86.97 -> 87.65 | Helped |
| Add depth modality (ablation) | UR-DMU · UCF-Crime | AUC: 86.97 -> 87.75 | Helped |
Surprising findings
- Reconstructing modalities at test time gives only a marginal ~0.25% gain, so the modality-free inference path is nearly as accurate.
- The depth modality alone gives the largest AUCA gain among single modalities, while motion gives the largest AUC gain.
Limitations
- Training requires five extra modality backbones, so training cost is high.
- Inference is slower than the RGB-only baseline (30.51 vs 110.09 FPS).
Experimental setup
Datasets: UCF-Crime XD-Violence MSAD
Benchmarks: UCF-Crime XD-Violence MSAD
Metrics: AUC AUCA AP APA FPS GFLOPs
Baselines: UR-DMU VadCLIP RTFM MGFN UMIL TSA TPWNG PEMIL MSL CLAV
Builds on: UR-DMU MIL ranking InfoNCE ViFiCLIP CLIP
Closest papers by meaning
- VadCLIP: Adapting Vision-Language Models for Weakly Supervised Video Anomaly Detection (AAAI 2024 · similarity 0.88)
- Learning Event Completeness for Weakly Supervised Video Anomaly Detection (ICML 2025 · similarity 0.88)
- Rethinking Reverse Distillation for Multi-Modal Anomaly Detection (AAAI 2024 · similarity 0.86)
- Harnessing Large Language Models for Training-free Video Anomaly Detection (CVPR 2024 · similarity 0.86)
- FrameShield: Adversarially Robust Video Anomaly Detection (NeurIPS 2025 · similarity 0.86)
- Federated Weakly Supervised Video Anomaly Detection with Multimodal Prompt (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.