Research map / Large language models / Agents, code and math
PANDA: Towards Generalist Video Anomaly Detection via Agentic AI Engineer
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PANDA, a training-free agentic MLLM framework for generalist video anomaly detection that plans scene-aware strategies via RAG, reasons with heuristic prompts, self-reflects with tools, and uses a chain-of-memory.
Problem
Video anomaly detection today relies on domain-specific training data or handcrafted pipelines, which limits generalization to new scenes and unseen anomaly types. The paper aims for VAD that works on any scene and anomaly type without training data or manual involvement.
Approach
An agentic AI engineer built on MLLMs that adaptively perceives the scene, retrieves anomaly rules via RAG to plan a detection strategy, performs goal-driven reasoning, reflects on uncertain ("Insufficient") cases by invoking tools, and accumulates short- and long-term chain-of-memory.
Key results
- Offline PANDA reaches 84.89% AUC on UCF-Crime, 70.16% AP on XD-Violence, 75.78% AUC on UBnormal and 73.12% AUC on CSAD, outperforming listed training-free baselines.
- Ablation on UCF-Crime: baseline 75.25% AUC rises to 84.89% with planning (+5.12), reflection (+2.26) and memory (+2.26).
- Holmes-VAU (instruction-tuned) reaches 88.96% UCF-Crime AUC and 87.68% XD AP, above PANDA on those two datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PANDA instead of Holmes-VAU | UCF-Crime · UCF-Crime | AUC (%): 84.89 vs 88.96 | Hurt |
| Use PANDA instead of Holmes-VAU | XD-Violence · XD-Violence | AP (%): 70.16 vs 87.68 | Hurt |
| Use PANDA instead of AnomalyRuler | UBnormal · UBnormal | AUC (%): 75.78 vs 71.90 | Helped |
| Use PANDA instead of Holmes-VAU | CSAD · CSAD | AUC (%): 73.12 vs 72.47 | Helped |
| Add planning to direct user-query reasoning (ablation) | PANDA on UCF-Crime · UCF-Crime | AUC (%): +5.12% | Helped |
| Add reflection to planning (ablation) | PANDA on UCF-Crime · UCF-Crime | AUC (%): +2.26% | Helped |
| Add memory to planning and reflection (ablation) | PANDA on UCF-Crime · UCF-Crime | AUC (%): +2.26% | Helped |
| Increase reflection rounds from 1 to 3 (ablation) | PANDA on UCF-Crime · UCF-Crime | AUC (%): 83.83 -> 84.89 | Helped |
| Increase reflection rounds from 3 to 5 (ablation) | PANDA on UCF-Crime · UCF-Crime | AUC (%): 84.89 -> 84.91 | Helped |
| Increase retrieved rules from 1 to 5 (ablation) | PANDA on UCF-Crime · UCF-Crime | AUC (%): 82.79 -> 84.89 | Helped |
| Increase retrieved rules from 5 to 9 (ablation) | PANDA on UCF-Crime · UCF-Crime | AUC (%): 84.89 -> 83.92 | Hurt |
| Increase short CoM length from 1 to 5 (ablation) | PANDA on UCF-Crime · UCF-Crime | AUC (%): 82.92 -> 84.89 | Helped |
Surprising findings
- Holmes-VAU slightly beats PANDA in online mode on CSAD, which the authors attribute to CSAD being partly drawn from Holmes-VAU's training sets.
- PANDA is below the supervised specialist methods on UCF-Crime and XD-Violence, trailing VadCLIP and Holmes-VAU.
Limitations
- Toolset is a curated fixed set and may not cover domain-specific modalities such as thermal imaging.
- Inference is slow (about 0.5 to 0.9 FPS) because of repeated MLLM calls and tool invocation during reflection.
- Detection is frame- or clip-level only, with no spatial localization of anomalies.
Experimental setup
Datasets: UCF-Crime XD-Violence UBnormal CSAD
Benchmarks: UCF-Crime XD-Violence UBnormal CSAD
Metrics: frame-level AUC average precision (AP)
Baselines: AED-MAE STPAG HL-Net RTFM UR-DMU VadCLIP TPWNG VERA Holmes-VAU ZS CLIP LLaVA-1.5 LAVAD
Builds on: retrieval-augmented generation chain-of-thought LAVAD AnomalyRuler Holmes-VAU
Closest papers by meaning
- Vad-R1: Towards Video Anomaly Reasoning via Perception-to-Cognition Chain-of-Thought (NeurIPS 2025 · similarity 0.88)
- Holmes-VAU: Towards Long-term Video Anomaly Understanding at Any Granularity (CVPR 2025 · similarity 0.86)
- VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models (CVPR 2025 · similarity 0.86)
- Harnessing Large Language Models for Training-free Video Anomaly Detection (CVPR 2024 · similarity 0.86)
- Open-Vocabulary Video Anomaly Detection (CVPR 2024 · similarity 0.86)
- A Unified Reasoning Framework for Holistic Zero-Shot Video Anomaly Analysis (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.