Research map / Video / Video-language understanding
Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HIVAU-70k, a hierarchical (clip/event/video) video anomaly understanding instruction dataset built with a semi-automated LLM annotation engine, and Holmes-VAU, an InternVL2-2B model with an Anomaly-focused Temporal Sampler that concentrates frames on anomaly-rich regions.
Problem
Existing video anomaly understanding datasets and models provide annotations at only one temporal granularity and sample frames inefficiently, limiting comprehension of anomalies across short and long temporal scales in long videos.
Approach
Combine a hierarchical multi-granularity instruction dataset (clip, event, video levels) built with manual segmentation plus recursive LLM summarization, with an anomaly-score-driven density-aware frame sampler (cumulative-sum inverse sampling) feeding a LoRA-tuned multimodal LLM.
Key results
- Holmes-VAU achieves 87.68% AP on XD-Violence and 88.96% AUC on UCF-Crime, outperforming prior SOTA such as VadCLIP (84.51 AP, 88.02 AUC).
- On HIVAU-70k reasoning tests, Holmes-VAU (2B) reaches video-level BLEU 0.566 and CIDEr 1.437 versus 0.120/0.031 for LLaVA-Next-Video (7B) and 0.155/0.044 for QwenVL2 (7B) (BLEU/CIDEr shown as video-level reported values).
- ATS outperforms uniform and Top-K sampling at 8, 16 and 32 frames; at 16 frames video-level CIDEr is 1.437 for ATS vs 1.345 for uniform and 1.302 for Top-K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Holmes-VAU instead of the strongest listed baseline, VadCLIP | Video anomaly detection · XD-Violence | AP: 84.51 -> 87.68 | Helped |
| Use Holmes-VAU instead of the strongest listed baseline, VadCLIP | Video anomaly detection · UCF-Crime | AUC: 88.02 -> 88.96 | Helped |
| Use hierarchical instruction tuning on HIVAU-70k | InternVL2-2B · Clip-level anomaly reasoning | BLEU: 0.435 -> 0.913 | Helped |
| Use hierarchical instruction tuning on HIVAU-70k | InternVL2-2B · Event-level anomaly reasoning | BLEU: 0.101 -> 0.804 | Helped |
| Use hierarchical instruction tuning on HIVAU-70k | InternVL2-2B · Video-level anomaly reasoning | BLEU: 0.155 -> 0.566 | Helped |
| Add event-level training data to clip-level training (ablation) | Holmes-VAU instruction tuning · Video-level anomaly reasoning | BLEU: 0.351 -> 0.349 | No clear effect |
| Add video-level training data to clip-level training (ablation) | Holmes-VAU instruction tuning · Video-level anomaly reasoning | BLEU: 0.351 -> 0.522 | Helped |
| Add video-level training data to event-level training (ablation) | Holmes-VAU instruction tuning · Video-level anomaly reasoning | BLEU: 0.292 -> 0.505 | Helped |
| Add event-level data to clip- and video-level training (ablation) | Holmes-VAU instruction tuning · Video-level anomaly reasoning | BLEU: 0.522 -> 0.566 | Helped |
| Replace ATS with uniform sampling (ablation) | Holmes-VAU · Video-level anomaly reasoning | BLEU: 0.566 -> 0.511 | Hurt |
| Replace ATS with Top-K sampling (ablation) | Holmes-VAU · Video-level anomaly reasoning | BLEU: 0.566 -> 0.476 | Hurt |
| Increase sampled frames from 16 to 32 with ATS (ablation) | Holmes-VAU · Video-level anomaly reasoning | BLEU: 0.566 -> 0.576 | Helped |
Surprising findings
- Top-K anomaly-score sampling underperforms uniform sampling-adjacent ATS because it overfocuses on local anomaly frames and misses context.
Limitations
- Clip-level annotations are mostly captions; anomaly analysis is only available for event and video levels.
- Annotation relies on manual segmentation and manual checking, which limits scalability.
- Evaluation uses n-gram metrics (BLEU, CIDEr, METEOR, ROUGE) with no LLM or human judgment of reasoning quality.
Experimental setup
Datasets: UCF-Crime XD-Violence UCA
Benchmarks: UCF-Crime XD-Violence HIVAU-70k test set Video-MME
Metrics: AUC AP BLEU CIDEr METEOR ROUGE
Baselines: Video-ChatGPT Video-LLaMA Video-LLaVA LLaVA-Next-Video QwenVL2 InternVL2 LAVAD VadCLIP UR-DMU Uniform sampling Top-K sampling
Builds on: InternVL2 LAVAD UR-DMU LoRA
Closest papers by meaning
- VADTree: Explainable Training-Free Video Anomaly Detection via Hierarchical Granularity-Aware Tree (NeurIPS 2025 · similarity 0.91)
- Open-Vocabulary Video Anomaly Detection (CVPR 2024 · similarity 0.90)
- Harnessing Large Language Models for Training-free Video Anomaly Detection (CVPR 2024 · similarity 0.90)
- MoniTor: Exploiting Large Language Models with Instruction for Online Video Anomaly Detection (NeurIPS 2025 · similarity 0.89)
- HAWK: Learning to Understand Open-World Video Anomalies (NeurIPS 2024 · similarity 0.89)
- VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.