Research map / Video / Video-language understanding
Towards Surveillance Video-and-Language Understanding: New Dataset Baselines and Challenges
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces UCA, a sentence-level annotated version of the UCF-Crime surveillance dataset, benchmarks seven-plus methods across temporal sentence grounding, video captioning, dense video captioning, and multimodal anomaly detection, and shows a surveillance-domain captioner improves TEVAD.
Problem
Surveillance video datasets provide only anomaly category labels and timings, lacking sentence-level language annotations, so multimodal surveillance video-and-language understanding tasks (grounding, captioning, dense captioning) are understudied.
Approach
Extends UCF-Crime with fine-grained manually written event descriptions and start/end timestamps (UCA), then benchmarks existing VALU models on it and uses a Swinbert captioner pretrained on UCA as a plug-in feature branch for multimodal anomaly detection.
Key results
- On UCA, the best TSGV method (MMN) reaches only 8.68% R@1 at IoU=0.3, with all methods below 10%.
- Enhanced TEVAD with a surveillance-domain Swinbert captioner reaches 85.3% AUC on UCF-Crime, versus 84.9% for TEVAD and 83.1% for visual features only.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MMN instead of CTRL for temporal sentence grounding | UCA TSGV · UCA | R@1, IoU=0.3: 6.35 -> 8.68 | Helped |
| Use CoCap instead of S2VT with Inception V4 features | UCA video captioning · UCA | CIDEr: 15.44 -> 21.24 | Helped |
| Use PDVC instead of TDA-CG with I3D features | UCA dense video captioning · UCA, predicted proposals | CIDEr: 5.51 -> 8.78 | Helped |
| Add surveillance-domain SwinBERT captions | Enhanced TEVAD · UCF-Crime | AUC: 84.9% -> 85.3% | Helped |
Surprising findings
- Mainstream models designed for conventional video datasets perform poorly on surveillance video, with all TSGV methods below 10% R@1 at IoU=0.3.
Limitations
- Dataset is limited in volume, built on only 1,854 UCF-Crime videos.
- Surveillance videos have low resolution and blur, making visual-language alignment harder.
- Annotation is text-only with no gender-neutral variant used in experiments.
Experimental setup
Datasets: UCF-Crime UCA VATEX
Benchmarks: UCA UCF-Crime
Metrics: R@1 at IoU 0.3/0.5/0.7 R@5 BLEU-1..4 METEOR ROUGE-L CIDEr recall and precision at IoU 0.3-0.9 SODA_c micro-averaged AUC
Baselines: CTRL SCDM A2C 2D-TAN LGI MMN MomentDiff S2VT RecNet MARN SGN SwinBERT
Builds on: UCF-Crime TEVAD SwinBERT PDVC
Closest papers by meaning
- Harnessing Large Language Models for Training-free Video Anomaly Detection (CVPR 2024 · similarity 0.86)
- Open-Vocabulary Video Anomaly Detection (CVPR 2024 · similarity 0.85)
- Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity (CVPR 2025 · similarity 0.84)
- MoniTor: Exploiting Large Language Models with Instruction for Online Video Anomaly Detection (NeurIPS 2025 · similarity 0.84)
- Ex-VAD: Explainable Fine-grained Video Anomaly Detection Based on Visual-Language Models (ICML 2025 · similarity 0.84)
- Learning Event Completeness for Weakly Supervised Video Anomaly Detection (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.