Research map / Vision-language and multimodal / Multimodal LLMs
VLForgery Face Triad: Detection, Localization and Attribution via Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLForgery, which fine-tunes LLaVA-1.5-7B with LoRA on a new diffusion-generated face dataset (VLF) and chain-of-thought descriptions (EkCot) to perform detection, localization, and attribution of forged faces.
Problem
Diffusion-model-generated faces are hard to detect, and existing detectors give only binary real/fake decisions without localizing partially synthesized regions or attributing the generator; prior MLLM forensics work focuses on traditional face-swap deepfakes.
Approach
Combines an extrinsic-knowledge-guided chain-of-thought (EkCot) that injects generation-pipeline metadata and low-level vision discrepancy phrases (from a VLLM comparison of real vs fake images) into forgery descriptions, then fine-tunes an MLLM to answer detection, localization and attribution in a structured sequence.
Key results
- VLForgery reaches 93.82% average cross-generator detection accuracy when trained on SDXL, versus 87.07% for LLaVA-1.5-7B and 88.55% for CLIPping linear probing.
- Partial-synthesis detection on SDXL improves to 78.62% with EkCot+visual features, compared with 69.92% for the baseline LLaVA-based variant.
- Attribution averages 82.03% on VLF, with full-synthesis attribution around 91.71% average and partial-synthesis attribution below 70%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add EkCot to forgery description generation (ablation) | VLForgery · VLF detection (partial synthesis, SDXL test) | accuracy: 68 -> 78.62 | Helped |
| Add visual input features to forgery description generation (ablation) | VLForgery · VLF detection (partial synthesis, SDXL test) | accuracy: 68 -> 78.62 | Helped |
| Add EkCot to forgery description generation (ablation) | VLForgery · VLF localization (hair, SDXL test) | accuracy: 88.77 -> 88.99 | No clear effect |
| Add visual input features to forgery description generation (ablation) | VLForgery · VLF localization (nose, SDXL test) | accuracy: 40.36 -> 77.25 | Helped |
| Use VLForgery instead of PSCC-Net | VLF localization · VLF localization (SD2 test, hair) | accuracy: 53.57 -> 86.91 | Helped |
| Use VLForgery instead of Guarnera et al. | VLF attribution · VLF | accuracy (Avg.): 80.22 -> 82.03 | Helped |
Surprising findings
- Qwen2.5-VL-7B performs near chance in cross-generator detection (14.02% average) despite being an MLLM.
- Training on Flux and testing on InstantID drops accuracy to 48.1%, and SD3-trained detectors reach only 21.93% on InstantID.
Limitations
- Partial-synthesis detection and attribution remain weak (attribution below 70%).
- Cross-subtype generalization is inconsistent, e.g., InstantID and SD3 subsets transfer poorly.
- Localization is reported as natural-language region names evaluated partly by manual judgment on 1,000 samples.
Experimental setup
Datasets: CelebAMask-HQ FFHQ FF++ CelebA-HQ
Benchmarks: VLF
Metrics: accuracy
Baselines: Xception ResNet-50 SPSL F3Net SRM NPR GramNet SAFE CNNspot DRCT CLIPping LLaVA-1.5-7B
Builds on: LLaVA LoRA CLIP DRCT
Closest papers by meaning
- Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector (CVPR 2025 · similarity 0.91)
- A Hitchhiker's Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning (NeurIPS 2024 · similarity 0.90)
- Identity-Aware Vision-Language Model for Explainable Face Forgery Detection (AAAI 2026 · similarity 0.89)
- Towards General Visual-Linguistic Face Forgery Detection (CVPR 2025 · similarity 0.88)
- DF40: Toward Next-Generation Deepfake Detection (NeurIPS 2024 · similarity 0.87)
- Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.