Research map / Vision-language and multimodal / Visual question answering
A Hitchhiker's Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Converts face forgery detection into a VQA benchmark with binary, multiple-choice, and open-ended multi-label evaluation of zero-shot VLLMs (BLIP-2, InstructBLIP, LLaVA-1.5, etc.) across seven datasets, plus a LoRA-tuned CLIP vision encoder experiment.
Problem
Existing deepfake detection work uses binary discriminative classifiers and a single-region VQA dataset; there is no fair protocol to evaluate VLLMs on binary, multi-label fine-grained manipulation identification, especially open-ended generation.
Approach
A multi-stage VQA evaluation protocol for face forgery detection that tests binary prompts with synonyms, multiple-choice fine-grained labels, and open-ended responses scored by exact match, substring containment, or CLIP-text cosine similarity, without human captioning.
Key results
- In open-ended fine-grained evaluation with CLIP matching, LLaVA-1.5 reaches F1 74.1 on SeqDeepFake attributes and 71.7 on components, and CLIP matching improves F1 over contains matching.
- Fine-tuning the LLaVA-1.5 CLIP vision encoder with LoRA raises binary F1 by about 3.7 points on FF++ (76.83 vs 73.10).
- No VLLM clearly outperforms others; BLIP-2 is the most robust binary model despite being the smallest, and all models struggle on in-the-wild datasets such as Celeb-DF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the LlaVa-1.5 vision encoder on FF++ (ablation) | LlaVa-1.5 binary detection · FF++ | F1: 73.10 -> 76.83 (+3.73%) | Helped |
| Fine-tune the LlaVa-1.5 vision encoder on FF++ (ablation) | LlaVa-1.5 binary detection · SeqDeepFake attributes | F1: 66.15 -> 68.03 (+1.88%) | Helped |
| Fine-tune the LlaVa-1.5 vision encoder on FF++ (ablation) | LlaVa-1.5 binary detection · SeqDeepFake components | F1: 90.57 -> 90.20 (-0.38%) | Hurt |
| Fine-tune the LlaVa-1.5 vision encoder on FF++ (ablation) | LlaVa-1.5 binary detection · R-Splicer | F1: 92.50 -> 92.62 (+0.12%) | Helped |
| Fine-tune the LlaVa-1.5 vision encoder on FF++ (ablation) | LlaVa-1.5 binary detection · DFDC | F1: 58.24 -> 58.65 (+0.41%) | Helped |
| Fine-tune the LlaVa-1.5 vision encoder on FF++ (ablation) | LlaVa-1.5 binary detection · CelebDF | F1: 41.81 -> 43.53 (+1.72%) | Helped |
| Fine-tune the LlaVa-1.5 vision encoder on FF++ (ablation) | LlaVa-1.5 binary detection · DFW | F1: 61.56 -> 61.90 (+0.34%) | Helped |
| Fine-tune the LlaVa-1.5 vision encoder on FF++ (ablation) | LlaVa-1.5 binary detection · StyleGAN2 | F1: 38.62 -> 39.72 (+1.10%) | Helped |
| Fine-tune the LlaVa-1.5 vision encoder on FF++ (ablation) | LlaVa-1.5 binary detection · StyleGAN3 | F1: 45.66 -> 45.74 (+0.08%) | Helped |
| Use CLIP-distance matching instead of contains matching | Open-ended fine-grained VQA · SeqDeepFake attributes, SeqDeepFake components, and R-Splicer | F1 | Helped |
| Use CLIP-distance matching instead of contains matching | Open-ended fine-grained VQA · SeqDeepFake attributes, SeqDeepFake components, and R-Splicer | mAP | Hurt |
| Use the larger T5-xxl language generator instead of the base T5 generator | InstructBLIP binary detection · Binary detection benchmarks | performance | Hurt |
Surprising findings
- BLIP-2, the smallest model, is more robust to prompt instruction in the binary task than larger models.
- CLIP baseline performs near chance on several datasets, contrary to the expectation that CLIP retrieval is an upper bound.
Limitations
- All VLLMs evaluated zero-shot and fall well below purpose-built discriminative detectors, especially on in-the-wild data.
- Lack of fine-grained, language-described deepfake datasets restricts qualitative and fine-grained evaluation; GPT-4V results use a 5k-sample subset and training data is unknown.
- Multiple-choice setting yields many false positives and 'All of them'/'None of them' answers that hamper matching.
Experimental setup
Datasets: FF++ DFDC Celeb-DF WildDeepfake StyleGAN2 StyleGAN3 SeqDeepFake R-Splicer
Benchmarks: FF++ DFDC Celeb-DF WildDeepfake StyleGAN2 StyleGAN3 SeqDeepFake R-Splicer MagicBrush
Metrics: accuracy AUC F1-score mAP BERTScore human score
Baselines: CLIP GPT-4V Xception UCF X-Ray LAA-Net CORE REECE
Builds on: CLIP BLIP-2 InstructBLIP LLaVA LoRA
Closest papers by meaning
- Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector (CVPR 2025 · similarity 0.93)
- Identity-Aware Vision-Language Model for Explainable Face Forgery Detection (AAAI 2026 · similarity 0.92)
- VLForgery Face Triad: Detection, Localization and Attribution via Multimodal Large Language Models (NeurIPS 2025 · similarity 0.90)
- Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection (ICML 2025 · similarity 0.90)
- Towards General Visual-Linguistic Face Forgery Detection (CVPR 2025 · similarity 0.89)
- Veritas: Generalizable Deepfake Detection via Pattern-Aware Reasoning (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.