Research map / Vision-language and multimodal / Multimodal LLMs
Towards General Visual-Linguistic Face Forgery Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLFFD, which generates mixed forgery images with sentence-level prompts (PFIG) and jointly trains CLIP image/text encoders with coarse and fine-grained contrastive losses (C2F) for generalizable, interpretable face forgery detection.
Problem
Face forgery (deepfake) detectors trained with binary labels or masks generalize poorly to unseen manipulations and offer little interpretability, and existing datasets lack fine-grained text annotations.
Approach
Automatically derive fine-grained annotations (forgery region and type) by comparing real/fake pairs, synthesize mixed forgery images via blending, and train CLIP-based dual encoders with coarse ("this is a fake/real person") and fine-grained template prompts so that matching text explains the detection.
Key results
- Trained on FF++ (HQ), the method reaches 94.79 AUC on DFD, 84.74 on DFDC-P, 83.55 on Wild Deepfake and 84.80 on Celeb-DF, outperforming DCL on the unseen sets.
- Ablation on Celeb-DF: adding multimodal learning and PFIG raises AUC from 77.62 (baseline) to 84.80.
- Cross-manipulation (DF to FS) AUC rises substantially over DCL, e.g. FS-trained model on DF reaches 92.98 vs DCL 74.80.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VLFFD instead of DCL | face forgery detection trained on FF++ (HQ) · DFDC-P | AUC: 76.71 -> 84.74 | Helped |
| Use VLFFD instead of DCL | face forgery detection trained on FF++ (HQ) · Wild Deepfake | AUC: 71.14 -> 83.55 | Helped |
| Use VLFFD instead of UIA-ViT | face forgery detection trained on FF++ (HQ) · DFD | AUC: 94.68 -> 94.79 | Helped |
| Use VLFFD instead of UIA-ViT | face forgery detection trained on FF++ (HQ) · Celeb-DF | AUC: 82.41 -> 84.80 | Helped |
| Add multimodal learning to the CLIP image encoder without PFIG (ablation) | VLFFD · Celeb-DF | AUC: 77.62 -> 80.79 | Helped |
| Add PFIG without multimodal learning (ablation) | VLFFD · Celeb-DF | AUC: 77.62 -> 77.71 | No clear effect |
| Add PFIG to multimodal learning (ablation) | VLFFD · Celeb-DF | AUC: 80.79 -> 84.80 | Helped |
| Replace ImageNet pretraining with CLIP pretraining for ViT-L (ablation) | VLFFD · Wild Deepfake | AUC: 76.49 -> 84.80 | Helped |
| Replace mask supervision with text supervision using region and type (ablation) | VLFFD · Celeb-DF | AUC: 77.70 -> 84.80 | Helped |
| Replace digital region labels with text region supervision (ablation) | VLFFD · Celeb-DF | AUC: 79.73 -> 82.26 | Helped |
| Use Coarse-and-Fine instead of Coarse-Only training (ablation) | VLFFD · Celeb-DF | AUC: 80.79 -> 84.80 | Helped |
| Use Fine-and-Fine instead of Coarse-and-Fine training (ablation) | VLFFD · Celeb-DF | AUC: 84.80 -> 82.19 | Hurt |
Surprising findings
- Mask supervision alone generalizes worse than digital labels in the ablation, which the authors attribute to overfitting.
Limitations
- Fine-grained prompts depend on heuristic thresholds for region and type decisions chosen empirically.
- Multimodal LLM integration is qualitative only, trained with limited resources and no quantitative evaluation.
- Blend-boundary type is assigned by blending method choice rather than detected from the image.
Experimental setup
Datasets: FaceForensics++ DFDC-P DFD Celeb-DF Wild Deepfake GID-DF GID-F2F
Benchmarks: FaceForensics++ DFDC-P DFD Celeb-DF Wild Deepfake GID-DF GID-F2F
Metrics: AUC EER ACC
Baselines: Xception EfficientNet-B4 Face X-ray F3-Net MAT GFF LTW LRL DCL PCL+I2G SBI UIA-ViT
Builds on: CLIP DCL SBI Gram-Net MiniGPT-4
Closest papers by meaning
- Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector (CVPR 2025 · similarity 0.90)
- Identity-Aware Vision-Language Model for Explainable Face Forgery Detection (AAAI 2026 · similarity 0.89)
- A Hitchhiker's Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning (NeurIPS 2024 · similarity 0.89)
- VLForgery Face Triad: Detection, Localization and Attribution via Multimodal Large Language Models (NeurIPS 2025 · similarity 0.88)
- Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models (CVPR 2025 · similarity 0.88)
- ForgerySleuth: Empowering Multimodal Large Language Models for Image Manipulation Detection (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.