Research map / Video / Video segmentation and tracking
Noise-Resistant Video Anomaly Detection via RGB Error-Guided Multiscale Predictive Coding and Dynamic Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a noise-resistant unsupervised video anomaly detector that chains RGB-error-guided multiscale predictive coding for next-frame prediction with a U-Net whose memory modules (sparse aggregation, selective test-time update) reconstruct predicted frames.
Problem
Unsupervised video anomaly detection is prone to false and missed alarms because background noise interferes with learning object-level appearance and motion features, and because normal pattern diversity is poorly captured by memory modules that are only updated during training.
Approach
Predict the next frame from RGB error maps (ground truth minus prediction) using multiscale spatio-temporal refinement modules with channel/spatial attention and dilated convolutions, then reconstruct the predicted frame with independent dynamic memory modules that use sparse aggregation and a selective update rule applied during both training and testing.
Key results
- Object-level variant reaches 92.9% AUC on Avenue, 86.0% on ShanghaiTech and 82.1% on UCF-Crime, improving over MGSTRL by 0.5, 0.9 and 1.5 points.
- Frame-level variant reaches 98.7% on Ped2, 92.1% on Avenue, 84.3% on ShanghaiTech and 79.9% on UCF-Crime at 21 fps.
- Ablation shows RGB error input beats raw frame input (Avenue AUC 92.9 vs 90.8; ShanghaiTech 86.0 vs 83.2) and sparse aggregation beats full aggregation on ShanghaiTech (86.0 vs 83.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use object-level inputs | Ours · Avenue | AUC: 92.1 -> 92.9 | Helped |
| Use object-level inputs | Ours · ShanghaiTech | AUC: 84.3 -> 86.0 | Helped |
| Use object-level inputs | Ours · UCF-Crime | AUC: 79.9 -> 82.1 | Helped |
| Use object-level inputs | Ours · UCSD Ped2 | AUC: 98.7 -> 99.1 | Hurt |
| Add channel attention to stacked ConvLSTMs (ablation) | EG-MPC · Avenue | AUC: 89.6 -> 91.0 | Helped |
| Add spatial attention with dilated convolution to stacked ConvLSTMs (ablation) | EG-MPC · ShanghaiTech | AUC: 81.8 -> 84.7 | Helped |
| Use RGB error maps instead of frames as input (ablation) | EG-MPC · Avenue | AUC: 90.8 -> 92.9 | Helped |
| Use RGB error maps instead of frames as input (ablation) | EG-MPC · ShanghaiTech | AUC: 83.2 -> 86.0 | Helped |
| Set up independent DMMs for different feature levels (ablation) | DMM reconstruction network · Avenue | AUC: 91.5 -> 92.9 | Helped |
| Use sparse rather than full aggregation of memory items (ablation) | DMM reconstruction network · Avenue | AUC | Helped |
| Update memory items during testing as well as training (ablation) | DMM reconstruction network · Avenue | AUC | Helped |
| Combine next-frame prediction and predicted-frame reconstruction (ablation) | Ours · ShanghaiTech | AUC: 81.6 -> 86.0 | Helped |
Surprising findings
- The method underperforms HF2-VAD on UCSD Ped2 at object level, which the authors attribute to low resolution and grayscale frames making RGB-error motion capture less effective than optical flow.
- Updating memory at test time is needed for the best result, with the gain most pronounced on Avenue, which has less training data.
Limitations
- Object-level results depend on YOLOv3 preprocessing, and frame-level results are lower on complex datasets.
- Weaker on low-resolution grayscale data such as UCSD Ped2 where RGB-error motion features are less effective than optical flow.
Experimental setup
Datasets: UCSD Ped2 Avenue ShanghaiTech UCF-Crime
Benchmarks: UCSD Ped2 Avenue ShanghaiTech UCF-Crime
Metrics: AUC score gap (ΔS) fps
Baselines: MemAE NM-GAN ITAE LERF HSC SDMAE FFP AnoPCN MNAD APC BDPN FPDM
Builds on: ConvLSTM MemAE U-Net MNAD AnoPCN
Closest papers by meaning
- MULDE: Multiscale Log-Density Estimation via Denoising Score Matching for Video Anomaly Detection (CVPR 2024 · similarity 0.87)
- Multi-Scale Video Anomaly Detection by Multi-Grained Spatio-Temporal Representation Learning (CVPR 2024 · similarity 0.86)
- Video Anomaly Detection with Motion and Appearance Guided Patch Diffusion Model (AAAI 2025 · similarity 0.86)
- Dual Conditioned Motion Diffusion for Pose-Based Video Anomaly Detection (AAAI 2025 · similarity 0.85)
- Just Dance with pi! A Poly-modal Inductor for Weakly-supervised Video Anomaly Detection (CVPR 2025 · similarity 0.85)
- A Diffusion-Based Framework for Multi-Class Anomaly Detection (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.