Research map / Vision-language and multimodal / Human and embodied modeling
Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M2F2-Det, a CLIP-based deepfake detector with forgery prompt learning and a bridge adapter that jointly outputs a fake score and LLM-generated textual explanations.
Problem
Deepfake detectors output either binary scores or textual explanations but not both, and prior CLIP-based detectors lack effective forgery-specific prompts and do not integrate CLIP's open-set recognition with LLMs for explanation.
Approach
Forgery Prompt Learning (universal forgery prompts plus layer-wise forgery tokens in the CLIP text encoder) produces forged attention maps, a Bridge Adapter fuses CLIP image features with an EfficientNet-B4 deepfake encoder, and a frequency token from the detector conditions a LLaVA-style LLM (Vicuna-7B with LoRA) to generate explanations.
Key results
- On FF++ (c40), M2F2-Det reaches 93.83% accuracy and 96.58% AUC, beating TALL's second-best AUC by 2.01 points.
- On DD-VQA explanation judgment, M2F2-Det achieves 95.23% accuracy and 96.61 F1, 7.74 points above DDVQA-BLIP in accuracy.
- Inter-dataset training on FF++ gives 87.80 AUC on DFDC and 88.70 on FFIW, outperforming FreqBlender by 0.24 and 2.56 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the full M2F2-Det detector with Forgery Prompt Learning and Bridge Adapter | M2F2-Det · FF++ (c40) | AUC: 91.03 -> 96.58 | Helped |
| add UF-prompts (ablation) | EfficientNet-B4 baseline · FF++ (c40) | AUC: 91.03 -> 92.57 | Helped |
| add UF-prompts (ablation) | EfficientNet-B4 baseline · Celeb-DF | AUC: 65.78 -> 67.37 | Helped |
| add layer-wise forgery tokens (ablation) | EfficientNet-B4 baseline · FF++ (c40) | AUC: 91.03 -> 92.66 | Helped |
| add layer-wise forgery tokens (ablation) | EfficientNet-B4 baseline · Celeb-DF | AUC: 65.78 -> 66.08 | Helped |
| add UF-prompts and layer-wise forgery tokens together (ablation) | EfficientNet-B4 baseline · FF++ (c40) | AUC: 91.03 -> 93.65 | Helped |
| add UF-prompts and layer-wise forgery tokens together (ablation) | EfficientNet-B4 baseline · Celeb-DF | AUC: 65.78 -> 68.68 | Helped |
| add Bridge Adapter (ablation) | EfficientNet-B4 baseline · FF++ (c40) | AUC: 91.03 -> 93.80 | Helped |
| add Bridge Adapter (ablation) | EfficientNet-B4 baseline · Celeb-DF | AUC: 65.78 -> 70.71 | Helped |
| add Bridge Adapter with UF-prompts (ablation) | EfficientNet-B4 baseline · FF++ (c40) | AUC: 91.03 -> 94.20 | Helped |
| add Bridge Adapter with UF-prompts (ablation) | EfficientNet-B4 baseline · Celeb-DF | AUC: 65.78 -> 71.08 | Helped |
| add Bridge Adapter, UF-prompts, and layer-wise forgery tokens (ablation) | EfficientNet-B4 baseline · Celeb-DF | AUC: 65.78 -> 74.82 | Helped |
Surprising findings
- Off-the-shelf LLaVA, Qwen and InternVL score far lower on DD-VQA judgment accuracy (42-59%) than the fine-tuned variants, and the zero-shot LLaVA result is only 51.41%.
- Removing the frequency token HF reduces M2F2-Det's DD-VQA judgment accuracy by about 10 points.
Limitations
- Inter-dataset result on DFD is weaker than AUNet, which the authors attribute to action-unit cues not modeled.
- Explanation evaluation uses keyword extraction for judgment, which may not capture full correctness.
- Detection is single-frame, so video temporal cues are not used.
Experimental setup
Datasets: FaceForensics++ DD-VQA Celeb-DF DFD DFDC WildDeepfake FFIW
Benchmarks: FaceForensics++ Celeb-DF WildDeepfake DFDC FFIW DFD DD-VQA
Metrics: AUC accuracy F1 BLEU-4 CIDEr ROUGE-L METEOR SPICE
Baselines: RFM Freq-SCL Add-Net F3-Net MultiAtt RECCE TALL DDVQA-BLIP LLaVA Qwen InternVL Uni-Fake
Builds on: CoOp CoCoOp LLaVA CLIP EfficientNet
Closest papers by meaning
- A Hitchhiker's Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning (NeurIPS 2024 · similarity 0.93)
- Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection (ICML 2025 · similarity 0.92)
- VLForgery Face Triad: Detection, Localization and Attribution via Multimodal Large Language Models (NeurIPS 2025 · similarity 0.91)
- Identity-Aware Vision-Language Model for Explainable Face Forgery Detection (AAAI 2026 · similarity 0.90)
- Towards General Visual-Linguistic Face Forgery Detection (CVPR 2025 · similarity 0.90)
- DF40: Toward Next-Generation Deepfake Detection (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.