Research map / Label-efficient and robust learning / Self-supervised and contrastive
FSFM: A Generalizable Face Security Foundation Model via Self-Supervised Facial Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FSFM, a self-supervised pretraining framework combining masked image modeling (with a CRFR-P facial-region masking strategy) and self-distillation instance discrimination on real faces, yielding a vanilla ViT-B foundation model for deepfake detection, face anti-spoofing, and diffusion forgery detection.
Problem
Face security tasks (deepfake detection, face anti-spoofing, diffusion facial forgery detection) generalize poorly; existing backbones are trained from scratch or ImageNet-supervised, and facial SSL methods lack "realness" representations needed for cross-dataset generalization.
Approach
Pretrain a ViT on unlabeled real faces with MAE-style masked reconstruction guided by a face-parser-based CRFR-P masking (cover a random facial region, proportionally mask the rest) jointly with a BYOL-style local-to-global self-distillation ID network, then fine-tune the vanilla ViT encoder for downstream face security tasks.
Key results
- FSFM ViT-B achieves average frame-level AUC gains of 15-20 points over ViT-B scratch and about 9 points over ImageNet-supervised ViT-B on cross-dataset deepfake detection trained on FF++.
- On cross-domain FAS (leave-one-out), FSFM ViT-B reaches average HTER 4.81% and AUC 98.68%, outperforming MCF (7.80%) and the DG-specialized GAC-FAS (6.52%).
- On DiFF, FSFM achieves 70.99 average AUC without FF++ subset, versus 44.21 for ImageNet-supervised ViT-B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain FSFM with MIM and ID on unlabeled real faces | ViT-B for cross-dataset DfD · Cross-dataset DfD | Video-level AUC: 82.06 -> 86.96 | Helped |
| Pretrain FSFM with MIM and ID on unlabeled real faces | ViT-B for cross-dataset DfD · Cross-dataset DfD | Frame-level AUC: 75.08 -> 85.26 | Helped |
| Use FSFM pretrained weights instead of supervised ImageNet initialization | ViT-B for cross-domain FAS · Cross-domain FAS | HTER: 6.56 -> 4.81 | Helped |
| Use FSFM pretrained weights instead of supervised ImageNet initialization | ViT-B for cross-domain FAS · Cross-domain FAS | AUC: 97.44 -> 98.68 | Helped |
| Use FSFM pretraining instead of training ViT-B from scratch | ViT-B for DiFF · DiFF (average w/o FF++) | AUC: 65.09 -> 70.99 | Helped |
| Replace random masking with CRFR-P facial masking (ablation) | MAE facial pretraining · DfD and FAS | Video-level AUC: 79.51 -> 81.58 | Helped |
| Replace the visible or masked ID target view with the full-face target view (ablation) | MAE with CRFR-P and ID · DfD and FAS | Video-level AUC: 81.50 -> 82.31; 81.86 -> 82.31 | Helped |
| Use 2-layer representation decoders in both branches instead of omitting them (ablation) | FSFM ID network · DfD and FAS | Video-level AUC: 81.48 -> 82.31; 81.14 -> 82.31 | Helped |
| Use 2-layer representation decoders instead of 1-layer or 3-layer decoders (ablation) | FSFM ID network · DfD and FAS | Video-level AUC: 80.68 -> 82.31; 80.71 -> 82.31 | Helped |
| Remove crop+flip augmentation from the online or target view (ablation) | FSFM pretraining · DfD and FAS | Video-level AUC: 81.54 -> 82.31; 78.80 -> 82.31 | Helped |
| Replace InfoNCE or MSE with the asymmetric cosine ID loss (ablation) | FSFM ID network · DfD and FAS | Video-level AUC: 80.60 -> 82.31; 79.34 -> 82.31 | Helped |
| Increase pretraining data from FF++ o to YTF or VF2 (ablation) | FSFM · DfD and FAS | Video-level AUC: 79.51 -> 83.86; 79.51 -> 87.89 | Helped |
Surprising findings
- Pretrained base models (ViT-B Sup(IN), MAE, DINO) perform worse than a ViT trained from scratch on diffusion forgery detection (DiFF), attributed to overfitting to the deepfake distribution.
- Facial SSL model MCF pretrained on 20M facial images generalizes worse to deepfake detection than natural-image MAE/DINO pretraining.
Limitations
- Gains are reported primarily as averaged metrics over downstream benchmarks; the best CDFv2 video-level result is held by synthetic-data methods.
- Pretraining relies on an off-the-shelf face parser to define facial regions.
Experimental setup
Datasets: VGGFace2 FaceForensics++ CelebA-Spoof MSU-MFSD CASIA-FASD Replay-Attack OULU-NPU DiFF
Benchmarks: CelebDF-v2 DFDC DFDCP WildDeepfake MSU-MFSD CASIA-FASD Replay-Attack OULU-NPU DiFF
Metrics: AUC HTER video-level AUC frame-level AUC
Baselines: ViT-B Scratch ViT-B Sup(IN) MAE DINO MCF SBIs NACO UDG-FAS GAC-FAS LAA-Net
Builds on: MAE BYOL DINO FaRL MARLIN
Closest papers by meaning
- Masked Face Recognition with Generative-to-Discriminative Representations (ICML 2024 · similarity 0.89)
- Self-Supervised Facial Representation Learning with Facial Region Awareness (CVPR 2024 · similarity 0.88)
- Guard Me If You Know Me: Protecting Specific Face-Identity from Deepfakes (NeurIPS 2025 · similarity 0.87)
- FaceShield: Explainable Face Anti-Spoofing with Multimodal Large Language Models (AAAI 2026 · similarity 0.87)
- SLIP: Spoof-Aware One-Class Face Anti-Spoofing with Language Image Pretraining (AAAI 2025 · similarity 0.87)
- PrefAce: Face-Centric Pretraining with Self-Structure Aware Distillation (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.