Research map / Vision-language and multimodal / CLIP and zero-shot
Standing on the Shoulders of Giants: Reprogramming Visual-Language Model for General Deepfake Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RepDFD, which reprograms a frozen CLIP ViT-L by learning universal visual input perturbations and sample-level face-embedding text prompts for generalizable deepfake detection with only 0.078M trainable parameters.
Problem
Deepfake face detectors generalize poorly to unseen datasets and manipulation methods, and full fine-tuning of large pre-trained models is costly and can erode pre-trained knowledge.
Approach
Applies model reprogramming to CLIP: only a learnable border visual prompt on the input image (Input Transformation) is optimized, while text prompts are made sample-specific by injecting face embeddings from a frozen face encoder (Face2Text), so that the text encoder guides the visual prompt optimization.
Key results
- Trained on FF++ (DF), the method reaches 88.41% AUC on Celeb-DF and 87.73% on WildDeepfake (frame-level), with 0.078M trainable parameters.
- Cross-manipulation: trained on FS, tested on DF gives 98.31% AUC vs 74.80% for DCL; trained on FSh, tested on DF gives 89.99% vs 63.98% for DCL.
- Compared with full fine-tuning of ViT-L (303M params), average AUC across CDF/Wild/DFDCP is 91.01% vs 81.30%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| reprogram frozen CLIP with learnable visual prompts and Face2Text prompts | CLIP-ViT-L · WildDeepfake, frame-level cross-dataset evaluation from FF++ | AUC (%): 83.55 -> 85.42 | Helped |
| reprogram frozen CLIP with learnable visual prompts and Face2Text prompts | CLIP-ViT-L · DFDCP, video-level cross-dataset evaluation from FF++ | AUC (%): 91.24 -> 95.03 | Helped |
| reprogram frozen CLIP with learnable visual prompts and Face2Text prompts | CLIP-ViT-L · DFDC, video-level cross-dataset evaluation from FF++ | AUC (%): 82.57 -> 80.99 | Hurt |
| use asymmetric face-conditioned prompts {T0,T3} instead of fixed prompts {T0,T1} (ablation) | RepDFD · Celeb-DF-v2, trained on FF++/DF | AUC (%): 82.88 -> 88.41 | Helped |
| use asymmetric face-conditioned prompts {T0,T3} instead of fixed prompts {T0,T1} (ablation) | RepDFD · WildDeepfake, trained on FF++/DF | AUC (%): 84.91 -> 87.73 | Helped |
| use asymmetric face-conditioned prompts {T0,T3} instead of fixed prompts {T0,T1} (ablation) | RepDFD · DFDCP, trained on FF++/DF | AUC (%): 88.62 -> 90.68 | Helped |
| replace fixed text prompts {T0,T1} with random frozen text embeddings (ablation) | RepDFD · WildDeepfake, trained on FF++/DF | AUC (%): 84.91 vs 82.30 | No clear effect |
| replace fixed text prompts {T0,T1} with random frozen text embeddings (ablation) | RepDFD · Celeb-DF-v2, trained on FF++/DF | AUC (%): 82.88 vs 82.46 | No clear effect |
| increase visual prompt width from p=34 to p=78 (ablation) | RepDFD · Cross-dataset average (CDF, WildDeepfake, DFDCP), trained on FF++/DF | AUC (%): 88.94 -> 77.06 | Hurt |
| replace visual prompting with full fine-tuning (ablation) | CLIP-ViT-L for deepfake detection · WildDeepfake, cross-dataset evaluation | AUC (%): 70.20 -> 88.05 | Helped |
| replace visual prompting with linear probing (ablation) | CLIP-ViT-L for deepfake detection · WildDeepfake, cross-dataset evaluation | AUC (%): 74.33 -> 88.05 | Helped |
| replace visual prompting with CoOp text prompting (ablation) | CLIP-ViT-L for deepfake detection · WildDeepfake, cross-dataset evaluation | AUC (%): 74.07 -> 88.05 | Helped |
Surprising findings
- Fixed or random text embeddings perform similarly, suggesting semantic content of text prompts matters little for detection; the gain comes from face-embedding-based sample-level prompts.
- Applying face embeddings only to the fake-class prompt (asymmetric setting) works best.
Limitations
- Moderate performance on Celeb-DF in cross-dataset evaluation relative to some baselines (e.g., LAA-Net, Bi-LIG at video level).
- Performance is sensitive to the visual prompt border width p; larger p degrades generalization due to information loss from downscaling.
- Random projection of face embeddings is needed for ViT-L dimension mismatch, and the method depends on a face encoder.
Experimental setup
Datasets: FaceForensics++ Celeb-DF-v2 DFDCP DFDC WildDeepfake
Benchmarks: FaceForensics++ Celeb-DF-v2 DFDCP DFDC WildDeepfake
Metrics: AUC EER
Baselines: DCL IID UIA-ViT CFM SLADD FoCus UCF TALL TALL++ SBI AUNet SeeABLE
Builds on: CLIP Visual Prompting (VP) AutoVP CoOp model reprogramming
Closest papers by meaning
- Unlocking the Capabilities of Large Vision-Language Models for Generalizable and Explainable Deepfake Detection (ICML 2025 · similarity 0.91)
- Veritas: Generalizable Deepfake Detection via Pattern-Aware Reasoning (ICLR 2026 · similarity 0.90)
- Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Model (CVPR 2025 · similarity 0.90)
- Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector (CVPR 2025 · similarity 0.89)
- D^3: Scaling Up Deepfake Detection by Learning from Discrepancy (CVPR 2025 · similarity 0.89)
- DF40: Toward Next-Generation Deepfake Detection (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.