Research map / Video / Video segmentation and tracking
On Learning Multi-Modal Forgery Representation for Diffusion Generated Video Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MM-Det, which combines multi-modal forgery representations from a fine-tuned LLaVA (LMM branch) with a VQ-VAE-reconstruction-based spatio-temporal Transformer (IAFA) and dynamic fusion to detect diffusion-generated videos, and introduces the DVF dataset.
Problem
Detecting diffusion-generated videos with diverse semantics, which existing video forgery detectors (mostly designed for facial deepfakes) and frame-level CNN/CLIP detectors fail to generalize to.
Approach
Use hidden states of a LoRA-instruction-tuned LLaVA as a multi-modal forgery representation (MMFR), combine it with a spatio-temporal ViT branch fed with VQ-VAE reconstruction residuals, using In-and-Across Frame Attention and a dynamic channel-weighting fusion.
Key results
- MM-Det achieves 92.0% average AUC on DVF, versus 84.3% for the second-best method HiFi-Net (+7.7%).
- The MMFR-only detector reaches 88.7% average AUC, 13.2 points above the ViT base model (75.5%).
- Replacing Vicuna-7b with Vicuna-13b yields 92.2% average AUC, a marginal gain over 92.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add diffusion reconstruction to Hybrid ViT (ablation) | Hybrid ViT video detector · DVF average | AUC (%): +1.7% AUC score | Helped |
| Add IAFA to Hybrid ViT with reconstruction (ablation) | Hybrid ViT video detector · DVF average | AUC (%): +13.0% AUC score | Helped |
| Use MMFR representation (ablation) | Video forgery detector · DVF average | AUC (%): +13.2% | Helped |
| Combine MMFR and ST branch (ablation) | MM-Det · DVF average | AUC (%): +0.4% | Helped |
| Add dynamic fusion (ablation) | MMFR + ST branch detector · DVF average | AUC (%): 89.1 {\scriptstyle \,\pm\, 2.3} -> 92.0 {\scriptstyle \,\pm\, 2.6} | Helped |
| Replace Vicuna-7b with Vicuna-13b (ablation) | MM-Det · DVF average | AUC (%): +0.2% | Helped |
| Replace Vicuna-7b with Mistral-7b (ablation) | MM-Det · DVF average | AUC (%): +2.2% higher for Vicuna-7b | Hurt |
| Add Vicuna-7b LLM to the N/A configuration (ablation) | MM-Det · DVF average | AUC (%): +3.5% | Helped |
| Use MM-Det instead of the strongest baseline | DVF detection · DVF average | AUC (%): 84.3 -> 92.0 | Helped |
Surprising findings
- Frequency-based HiFi-Net and CLIP-feature detectors (Raising, Uni-FD) perform poorly on most DVF generators, while a simple fine-tuned CNN (CNNDet) outperforms them.
Limitations
- Limited to fully synthesized videos; does not generalize to partial manipulations, possibly because small traces vanish after downsampling.
- Integrating a large language model is computationally expensive and unsuitable for resource-limited settings.
- Robustness degrades under Gaussian blur (down to 86.2% AUC) and other post-processing, and the paper notes possible failure on blurred or noisy inputs.
Experimental setup
Datasets: Diffusion Video Forensics (DVF) Rich Forgery Reasoning Dataset InternVid-10M YouTube-8M ImageNet
Benchmarks: DVF
Metrics: AUC
Baselines: CNNDet DIRE Uni-FD Raising F3Net ViViT TALL TS2-Net DE-FAKE HiFi-Net
Builds on: LLaVA Hybrid ViT VQ-VAE DIRE LoRA
Closest papers by meaning
- Generalizing Deepfake Video Detection with Plug-and-Play: Video-Level Blending and Spatiotemporal Adapter Tuning (CVPR 2025 · similarity 0.89)
- BusterX: MLLM-Powered AI-Generated Video Forgery Detection and Explanation (NeurIPS 2026 · similarity 0.87)
- DiffForensics: Leveraging Diffusion Prior to Image Forgery Detection and Localization (CVPR 2024 · similarity 0.86)
- FakeShield: Explainable Image Forgery Detection and Localization via Multi-modal Large Language Models (ICLR 2025 · similarity 0.86)
- Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models (CVPR 2025 · similarity 0.85)
- Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.