Research map / Vision-language and multimodal / Human and embodied modeling
Guard Me If You Know Me: Protecting Specific Face-Identity from Deepfakes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VIPGuard, a three-stage fine-tuned Qwen2.5-VL-7B framework (facial attribute learning, identity discrimination with cross-attention, and a learnable VIP token) for personalized, explainable face deepfake detection, plus the VIPBench benchmark of 22 identities and 80,080 images.
Problem
Deepfake detectors are mostly identity-agnostic and give little explanation, while existing identity-aware detectors use mainly global features; the paper targets detecting forgeries of specific VIP individuals whose authentic face images are available.
Approach
Reformulates personalized deepfake detection as fine-grained face recognition against a protected identity: an MLLM is first taught facial attributes, then trained to reason over face-pair differences using similarity scores and attributes, and finally adapted per VIP via a lightweight learnable VIP token (Yo'LLaVA-style) with the MLLM frozen.
Key results
- On VIPEval face swapping, VIPGuard reaches about 99.48 AUC / 0.81 EER on BlendFace, versus 91.87 / 17.61 for the best general detector (Effort) and 88.67 / 13.37 for ICT-Ref.
- On entire face synthesis detection, VIPGuard achieves 89.03% ACC on GPT-4o images, compared with 83.63% AUC for RECCE and 45.26 AUC for DiffID on GPT-4o.
- Ablation shows the full three-stage pipeline gives 98.23 average AUC versus 50.24 for the base Qwen2.5-VL-7B, and annotation-free Stage 3 training drops the average AUC only from 98.98 to 95.86.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VIPGuard instead of the strongest listed general or identity-aware detector | Deepfake detection on VIPEval · Face-swapping and entire-face synthesis methods | AUC / EER (%) | Helped |
| Train Stage 3 with images and textual annotations rather than images only (ablation) | VIPGuard Stage 3 · BlendFace | AUC (%): 98.45 -> 99.48 | Helped |
| Train Stage 3 with images and textual annotations rather than images only (ablation) | VIPGuard Stage 3 · InSwap | AUC (%): 92.91 -> 99.43 | Helped |
| Add user-specific customization (Stage 3) to the baseline model (ablation) | Qwen-2.5-VL-7B · BlendFace | AUC (%): 51.00 -> 71.96 | Helped |
| Add facial attribute learning (Stage 1) before user-specific customization (ablation) | VIPGuard with Stage 3 · BlendFace | AUC (%): 71.96 -> 96.50 | Helped |
| Add general identity discrimination (Stage 2) to facial attribute learning and user-specific customization (ablation) | VIPGuard with Stages 1 and 3 · BlendFace | AUC (%): 96.50 -> 99.48 | Helped |
| Use only images rather than images and annotations for Stage 3 training (ablation) | VIPGuard Stage 3 · Average across BlendFace, InSwap, Arc2Face, and PuLID | AUC (%): 98.98 -> 95.86 | Hurt |
| Use Adaptive VIPGuard automatic VIP-token selection instead of VIPGuard (ablation) | VIPGuard · VIPEval average | AUC (%): 98.23 -> 98.01 | Hurt |
| Replace the Qwen-2.5-VL-7B backbone with LLaMA-3.2-Vision-11B (ablation) | VIPGuard · Average across BlendFace, InSwap, Arc2Face, and PuLID | AUC (%): 98.23 -> 95.94 | Hurt |
| Replace TransFace with CosFace as the face-recognition similarity model (ablation) | VIPGuard · Average across BlendFace, InSwap, Arc2Face, and PuLID | AUC (%): 98.23 -> 88.37 | Hurt |
| Increase VIP token count beyond 32 to 64 or 128 (ablation) | VIPGuard Stage 3 · Multiple forgery types | AUC (%) | Hurt |
| Increase the number of authentic target-user images used for customization (ablation) | VIPGuard Stage 3 · Multiple forgery types | AUC (%) | Helped |
Surprising findings
- Stage 3 alone (user customization without general identity training) only reaches 71.96 AUC on BlendFace, showing the foundational face-understanding and discrimination stages are necessary.
- Gemini 2.5 Pro and GPT-4o are fairly weak on some forgery types (e.g., GPT-4o 46.70% ACC on ConsistentID), and the base Qwen2.5-VL-7B is near random (about 50% AUC).
Limitations
- Performance drops under large head pose variation (>50 degrees) and large age differences between training and test images.
- Requires several authentic reference images per VIP; the annotation-free variant loses some accuracy and one-shot results are weaker on CelebDF/DF40.
- Does not exploit audio, 3D, or temporal video cues.
Experimental setup
Datasets: LAION-Face CrossFaceID FaceID-6M CelebDF DF40
Benchmarks: VIPBench (VIPEval) CelebDF DF40
Metrics: AUC EER ACC
Baselines: Xception EfficientNet UCF ProDet RECCE CDFA RepDFD Effort DiffID ICT-Ref FFAA GPT-4o
Builds on: Yo'LLaVA LoRA Qwen2.5-VL
Closest papers by meaning
- FaceShield: Explainable Face Anti-Spoofing with Multimodal Large Language Models (AAAI 2026 · similarity 0.88)
- Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector (CVPR 2025 · similarity 0.88)
- FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learning (CVPR 2025 · similarity 0.87)
- DF40: Toward Next-Generation Deepfake Detection (NeurIPS 2024 · similarity 0.87)
- DiffusionFake: Enhancing Generalization in Deepfake Detection via Guided Stable Diffusion (NeurIPS 2024 · similarity 0.87)
- From Specificity to Generality: Revisiting Generalizable Artifacts in Detecting Face Deepfakes (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.