Research map / Video / Video segmentation and tracking
Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a parameter-efficient side-network decoder with spatial and temporal modules on a frozen CLIP ViT-L/14 image encoder, plus a Facial Component Guidance InfoNCE loss, for generalizable video deepfake detection trained on FF++.
Problem
Generalizable video-based deepfake detection on unseen forgery datasets, where full fine-tuning of foundation models overfits to generic regions and dataset-specific cues.
Approach
A layer-wise side-network decoder with separate spatial (cross-attention with learnable queries) and temporal (patch-temporal MHSA) modules adapts a frozen CLIP image encoder; the FCG loss contrastively aligns spatial queries with lips, skin, eyes and nose features mined via facial landmarks.
Key results
- Cross-dataset video-level AUROC averages 92.3% across CDF, DFDC, FSh, DFo and WDF after training on FF++, with the largest gains on CDF (95.0), DFDC (81.8) and WDF (87.2).
- With about 250K trainable parameters versus 58.3M for EVL and 303M for full fine-tuning, the method reaches 92.3 average AUROC against 83.9 (EVL) and 85.1 (FFT).
- Removing the spatial module drops the average by 5.3 points and removing FCG drops it by 1.4 points; with 50% of training data the method still averages 90.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the spatial module (ablation) | Ours · Cross-dataset average (CDF, DFDC, FSh, DFo, WDF) | video-level AUC (%): 87.0(-5.3) -> 92.3 | Helped |
| add the temporal module (ablation) | Ours · Cross-dataset average (CDF, DFDC, FSh, DFo, WDF) | video-level AUC (%): 89.0(-3.3) -> 92.3 | Helped |
| add FCG loss (ablation) | Ours · Cross-dataset average (CDF, DFDC, FSh, DFo, WDF) | video-level AUC (%): 90.9(-1.4) -> 92.3 | Helped |
| use focal loss instead of cross-entropy loss (ablation) | Ours · Cross-dataset average (CDF, DFDC, FSh, DFo, WDF) | video-level AUC (%): 91.2(-1.1) -> 92.3 | Helped |
| train on 50% rather than 100% of FF++ training data (ablation) | Ours · Cross-dataset average (CDF, DFDC, FSh, DFo, WDF) | video-level AUC (%): 92.3 -> 90.7 | Hurt |
| train on 10% rather than 100% of FF++ training data (ablation) | Ours · Cross-dataset average (CDF, DFDC, FSh, DFo, WDF) | video-level AUC (%): 92.3 -> 83.5 | Hurt |
| adapt CLIP with the proposed side-network framework rather than full fine-tuning | CLIP ViT-L/14 · Cross-dataset average (CDF, DFDC, FSh, DFo, WDF) | video-level AUC (%): 85.1 -> 92.3 | Helped |
Surprising findings
- Full fine-tuning of CLIP attends to arbitrary small regions and generalizes worse than the parameter-efficient side network.
- RealForensics beats the proposed method on in-domain FF++ leave-one-out evaluation, yet degrades sharply on unseen CDF, DFDC and WDF, suggesting dataset bias.
Limitations
- Sparse sampling of only 10 frames per clip may miss high-frequency artifacts.
- Less robust than RealForensics on in-domain FF++ perturbations (about 1.8 points larger average drop).
- Relies on facial landmark detection and alignment preprocessing.
Experimental setup
Datasets: FaceForensics++ LRW CelebDF-v2 DFDC FaceShifter DeeperForensics WildDeepfake
Benchmarks: CelebDF-v2 DFDC FaceShifter DeeperForensics WildDeepfake FaceForensics++
Metrics: video-level AUROC average AUROC
Baselines: Xception Face X-ray LipForensics FTCN SBI RealForensics AltFreezing TALL-Swin LAA-Net EVL VPT Linear probing
Builds on: CLIP EVL RealForensics VPT LoRA
Closest papers by meaning
- From Specificity to Generality: Revisiting Generalizable Artifacts in Detecting Face Deepfakes (NeurIPS 2025 · similarity 0.90)
- Generalizing Deepfake Video Detection with Plug-and-Play: Video-Level Blending and Spatiotemporal Adapter Tuning (CVPR 2025 · similarity 0.90)
- DF40: Toward Next-Generation Deepfake Detection (NeurIPS 2024 · similarity 0.90)
- Standing on the Shoulders of Giants: Reprogramming Visual-Language Model for General Deepfake Detection (AAAI 2025 · similarity 0.90)
- DiffusionFake: Enhancing Generalization in Deepfake Detection via Guided Stable Diffusion (NeurIPS 2024 · similarity 0.90)
- Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.