Research map / Vision-language and multimodal / Multimodal LLMs

Identity-Aware Vision-Language Model for Explainable Face Forgery Detection

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an identity-aware personalized VLM (LLaVA-1.6-13B with learnable identity tokens and a shallow-feature detection adapter) for explainable face forgery detection, evaluated on a new IDImage dataset.

Problem

Face forgery detectors miss identity-level semantic inconsistencies and generalize poorly to unseen manipulation methods, while existing VLM-based detectors rely on costly supervised fine-tuning that struggles to preserve per-identity characteristics.

Approach

Encode each target identity's appearance and behavior priors into 10 learnable identifier/soft tokens (no full SFT), and add a lightweight detection adapter that feeds shallow vision-encoder features to the LLM to preserve low-level artifacts.

Key results

• Achieves 94.25% accuracy and 94.08% F1 on IDImage, versus 89.08% accuracy for the best traditional detector (ESSP) and 83.03% for GPT-4o.
• Explanation human rating of 4.01/5 versus 3.87 for GPT-4o and 2.91 for Qwen2.5-VL (All).
• With only 5% of training data, F1 is 68.00%; performance rises steadily with more data.

Surprising findings

• Qwen2.5-VL with zero-shot prompting reaches 100% precision but only 12.24% recall.
• Adding human-written identity descriptions to GPT-4o raises accuracy only modestly, while single reference images give limited personalization cues.

Limitations

• Training requires about 1,000 images per identity for best performance; performance drops sharply with less data.
• Identity tokens are trained separately per individual, so each new identity requires its own training run.
• Evaluation limited to 20 public figures and five manipulation methods with a small human rating study.

Experimental setup

Datasets: IDImage SimSwap training forgeries PhotoMaker-generated forgeries

Benchmarks: IDImage

Metrics: accuracy precision recall F1 human rating of explanation quality (5-point)

Baselines: Meso4 Xception Recce UCF NPR ESSP Qwen2.5-VL LLaVA FFAA Yo'LLaVA GPT-4o

Builds on: LLaVA Yo'LLaVA FFAA FakeShield

Closest papers by meaning

• A Hitchhiker's Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning (NeurIPS 2024 · similarity 0.92)
• Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector (CVPR 2025 · similarity 0.90)
• Towards General Visual-Linguistic Face Forgery Detection (CVPR 2025 · similarity 0.89)
• Spot the Fake: Large Multimodal Model-Based Synthetic Image Detection with Artifact Explanation (NeurIPS 2025 · similarity 0.89)
• VLForgery Face Triad: Detection, Localization and Attribution via Multimodal Large Language Models (NeurIPS 2025 · similarity 0.89)
• Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models (CVPR 2025 · similarity 0.89)