Research map / Video / Video-language understanding
HAWK: Learning to Understand Open-World Video Anomalies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HAWK, an interactive video-LLM for open-world video anomaly understanding that adds an explicit optical-flow motion branch, a video-motion consistency loss, and a motion-language matching loss, plus a ~8,000-video anomaly description and QA dataset.
Problem
Existing video anomaly detection systems offer shallow semantic understanding and little user interaction, and existing anomaly datasets lack detailed language descriptions and question-answer data for open-world use.
Approach
Build a dual-branch video-LLM (appearance and motion branches on BLIP-2 encoders feeding LLaMA-2) where optical-flow-masked frames supply motion, a cosine consistency loss aligns compressed video and motion features, and a cross-entropy loss supervises motion-related verbs and entities extracted by dependency parsing.
Key results
- On anomaly video description generation, HAWK reaches BLEU-4 0.043 and GPT-guided reasonability/detail/consistency of 0.283/0.320/0.218, the best among the five baselines in each metric.
- On anomaly video question-answering, HAWK reaches BLEU-4 0.073 and GPT-guided reasonability/detail/consistency of 0.840/0.794/0.753, above VideoChat's 0.699/0.631/0.598.
- Ablations show that removing motion information lowers description BLEU-4 from 0.043 to 0.034 and reasonability from 0.283 to 0.253.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add motion branch and motion input (ablation) | HAWK · Anomaly video description generation | BLEU-1: 0.249 -> 0.270 | Helped |
| add video-motion consistency loss (ablation) | HAWK · Anomaly video description generation | BLEU-1: 0.249 -> 0.270 | Helped |
| add motion-language matching loss (ablation) | HAWK · Anomaly video description generation | BLEU-2: 0.135 -> 0.139 | Helped |
| add motion branch and motion input (ablation) | HAWK · Anomaly video question-answering | BLEU-1: 0.309 -> 0.319 | Helped |
| add video-motion consistency loss (ablation) | HAWK · Anomaly video question-answering | BLEU-1: 0.313 -> 0.319 | Helped |
| add motion-language matching loss (ablation) | HAWK · Anomaly video question-answering | BLEU-1: 0.316 -> 0.319 | Helped |
Limitations
- Hallucinations persist when motion extraction is erroneous
- Only video-level understanding; no streaming anomaly detection
- Dataset coverage limits true open-world generalization; no anomaly localization
Experimental setup
Datasets: WebVid UCF-Crime ShanghaiTech CUHK Avenue UCSD Ped1 UCSD Ped2 DoTA UBnormal
Benchmarks: anomaly video description generation (own dataset test split) anomaly video question-answering (own dataset test split)
Metrics: BLEU-1 BLEU-2 BLEU-3 BLEU-4 GPT-guided reasonability GPT-guided detail GPT-guided consistency
Baselines: Video-ChatGPT VideoChat Video-LLaMA LLaMA-Adapter Video-LLaVA
Builds on: BLIP-2 Video-LLaMA LLaMA-2 Farneback optical flow
Closest papers by meaning
- Open-Vocabulary Video Anomaly Detection (CVPR 2024 · similarity 0.89)
- Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity (CVPR 2025 · similarity 0.89)
- VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models (CVPR 2025 · similarity 0.88)
- Harnessing Large Language Models for Training-free Video Anomaly Detection (CVPR 2024 · similarity 0.88)
- MoniTor: Exploiting Large Language Models with Instruction for Online Video Anomaly Detection (NeurIPS 2025 · similarity 0.87)
- A Unified Reasoning Framework for Holistic Zero-Shot Video Anomaly Analysis (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.