Research map / Vision-language and multimodal / Multimodal LLMs
Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an LVLM-based deepfake detector that aligns image features with real/fake text descriptions via ImageBind to produce consistency maps for classification and localization, then feeds forgery prompt embeddings into a Vicuna-7B LLM (PandaGPT architecture) with LoRA for explainable, multi-turn detection.
Problem
LVLMs have general visual knowledge but lack the fine-grained sensitivity needed for deepfake detection, and existing detectors generalize poorly and do not provide interpretable multi-turn explanations.
Approach
A Knowledge-guided Forgery Detector computes correlations between ImageBind image features and text embeddings of pristine/deepfake descriptions to form consistency maps; a Forgery Prompt Learner converts the resulting segmentation map, forgery score, and consistency maps into prompt embeddings fed to the LLM, alternated with general VQA data to keep dialogue ability.
Key results
- KFD reaches 94.71 AUC on CDF2, 99.64 on DFD, 97.62 on CDF1, outperforming listed SOTA on several cross-dataset benchmarks.
- The LVLM variant (Ours-LVLM) achieves 97.11 AUC on FF++ and 92.39 average AUC across FF++, CDF2, DFD, CDF1, DFDCP, versus 67.39 for FAK-Owl.
- Removing the Forgery Prompt Learner drops FF++ LLM-based AUC from 97.11 to 63.42 (LoRA only) and DFDC from 68.77 to 55.11.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add FPL to LLM inference without LoRA (ablation) | Vicuna-7B deepfake LVLM · FF++, CDF2, and DFDC | AUC: 50.00 -> 96.59; 50.28 -> 89.25; 50.00 -> 67.65 | Helped |
| Add LoRA to FPL + LLM (ablation) | Vicuna-7B deepfake LVLM · FF++, CDF2, and DFDC | AUC: 96.59 -> 97.11; 89.25 -> 89.90; 67.65 -> 68.77 | Helped |
| Add LoRA to LLM inference without FPL (ablation) | Vicuna-7B deepfake LVLM · FF++, CDF2, and DFDC | AUC: 50.00 -> 63.42; 50.28 -> 62.53; 50.00 -> 55.11 | Helped |
| Add reference-based optimization (ablation) | KFD · CDF1, CDF2, and DFDCP | AUC: 96.18 -> 97.62; 92.77 -> 94.71; 91.73 -> 91.81 | Helped |
| Add reference-based optimization (ablation) | KFD · DFDC | AUC: 79.31 -> 79.12 | Hurt |
| Vary training images from 50 to 500 (ablation) | Ours-LVLM · CDF1, CDF2, DFD, DFDC, and DFDCP | AUC: CDF1: 77.78, 82.59, 84.53, 84.17; CDF2: 82.64, 81.74, 80.68, 83.31; DFD: 82.25, 86.34, 88.14, 88.17; DFDC: 70.49, 72.72, | Mixed |
| Replace Vicuna-7B with Llama-3.2-1B or Llama-3.2-3B (ablation) | Ours-LVLM · FF++, CDF1, DFD, and DFDC | AUC: Vicuna-7B: 97.11, 95.97, 84.26, 67.65; Llama-3.2-1B: 96.49, 95.64, 82.74, 65.66; Llama-3.2-3B: 97.08, 95.30, 82.80, 66.1 | Hurt |
| Train on CDF2 or DFDCP instead of FF++ (ablation) | Ours-LVLM · FF++, CDF2, DFDCP, and DFDC | AUC: FF++ training: 99.54, 94.32, 92.39, 77.03; CDF2 training: 97.95, 95.79, 86.36, 70.27; DFDCP training: 99.15, 85.17, 93.7 | Mixed |
| Alternately train on general VQA data for dialogue capability (ablation) | Ours-LVLM · Deepfake detection | AUC | Hurt |
Surprising findings
- Without the Forgery Prompt Learner, LLM-only fine-tuning yields near-chance AUC (about 50 on FF++, CDF2, DFDC), showing the forgery prompt is essential.
- Ours-LVLM is markedly slower (1.211 s/frame) than FAK-Owl (0.642 s/frame) and CADDM (0.026 s/frame).
Limitations
- Alternating training with general VQA data introduces domain gaps that can reduce forgery detection performance.
- Failure cases where localization is accurate but the LLM text description does not match the identified region.
- Relies on GPT-4-generated and human-validated descriptions and on a simulated blending pipeline rather than real forgeries for LLM training.
Experimental setup
Datasets: FaceForensics++ CDF1 CDF2 DFD DFDCP DFDC DF40 PandaGPT VQA dataset
Benchmarks: FF++ CDF1 CDF2 DFD DFDC DFDCP DF40
Metrics: AUC AP
Baselines: Xception F3Net SBI RECCE CADDM UCF LAANet ProDet RepDFD UDD PandaGPT Qwen2-VL
Builds on: PandaGPT ImageBind SBI FAKE-Owl
Closest papers by meaning
- Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector (CVPR 2025 · similarity 0.92)
- Standing on the Shoulders of Giants: Reprogramming Visual-Language Model for General Deepfake Detection (AAAI 2025 · similarity 0.91)
- Veritas: Generalizable Deepfake Detection via Pattern-Aware Reasoning (ICLR 2026 · similarity 0.90)
- A Hitchhiker's Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning (NeurIPS 2024 · similarity 0.90)
- Exposing the Deception: Uncovering More Forgery Clues for Deepfake Detection (AAAI 2024 · similarity 0.89)
- Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Model (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.