Research map / Label-efficient and robust learning / Recognition and long-tail
Contrastive Learning for DeepFake Classification and Localization via Multi-Label Ranking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a unified deepfake detector that combines a contrastive multiple-instance learning loss for binary real/fake classification with a multi-label ranking loss that recovers which facial components were manipulated and in what order.
Problem
Binary deepfake classification is saturated, while sequential component-wise deepfake manipulation requires localizing which facial components were forged and recovering the order of manipulations.
Approach
Treats each image as a bag of patch-token similarity responses under a contrastive MIL loss for real/fake detection, and adds a rank-aware weighted multi-label loss in a CNN-transformer with class tokens so that predicted probabilities respect the manipulation order.
Key results
- On Seq-FaceComp with Swin Transformer backbone, multi-label accuracy 90.74% and ranking Fixed-Acc 70.02% (ResNet-50 ranking 69.58% vs. SeqFakeFormer 68.86%).
- Cross-testing trained on FF++ reaches 91.56% AUC on Celeb-DF versus 89.12% for SBIs (Swin backbone) and 75.17% on DFDC.
- Correlation R^2 between predicted probabilities and manipulation order rises from 0.35-0.46 (Multi-Cls) to 0.61-0.75 for the proposed method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-label ranking for component predictions | Ours · Seq-FaceComp | Ranking accuracy (%): 73.93 -> 74.97 | Helped |
| Use multi-label ranking for component predictions | Ours · Seq-FaceAttr | Ranking accuracy (%): 69.27 -> 70.02 | Helped |
| Use the proposed model | Seq-FaceComp · Seq-FaceComp | Multi-label accuracy (%): 79.54 -> 84.36 | Helped |
| Use the proposed model | Seq-FaceAttr · Seq-FaceAttr | Multi-label accuracy (%): 88.23 -> 90.74 | Helped |
| Use the proposed model | binary deepfake classification · CDF cross-testing (train on FF++ only) | AUC (%): 93.18 -> 91.56 | Hurt |
| Use the proposed model | binary deepfake classification · DFDC cross-testing (train on FF++ only) | AUC (%): 80.92 -> 75.17 | Hurt |
| Use the proposed model | binary deepfake classification · FF++ intra-testing | AUC (%): 96.30 -> 99.82 | Helped |
| Use the proposed model | binary deepfake classification · CDF intra-testing | AUC (%): 99.94 -> 99.98 | Helped |
| Use the proposed model | binary deepfake classification · WDF cross-testing (train on FF++ only) | AUC (%): 70.56 -> 73.41 | Helped |
| Use the proposed model | binary deepfake classification · DFD cross-testing (train on FF++ only) | AUC (%): 97.56 -> 97.88 | Helped |
| Add the ranking loss LRank (ablation) | multi-label predictions on Seq-FaceComp · Seq-FaceComp | Fixed-Acc (%) | Helped |
| Add the contrastive MIL loss LCLS (ablation) | the proposed model on Seq-FaceComp · Seq-FaceComp | Fixed-Acc (%) | Helped |
Surprising findings
- Intra-dataset FF++ AUC is nearing a plateau (~99.8%), so the authors argue cross-dataset generalization is the principal remaining obstacle.
Limitations
- Sequential-manipulation experiments are limited to the Seq-DeepFake datasets with a fixed set of facial components.
- Brute-force evaluation of k over all similarity entries is computationally prohibitive, requiring approximate sampling.
Experimental setup
Datasets: Seq-FaceComp Seq-FaceAttr FaceForensics++
Benchmarks: Seq-FaceComp Seq-FaceAttr FaceForensics++ Celeb-DF WildDeepfake DeepFakeDetection DFDC
Metrics: Fixed-Acc AUC R^2
Baselines: Multi-Cls DRN DETR MA Two-Stream SeqFakeFormer MMNet Xception EfficientNet-B4 Multi-Att SPSL RECCE
Builds on: SeqFakeFormer Multiple instance learning attention pooling Differentiable sorting networks Self-blended images
Closest papers by meaning
- Exploring Unbiased Deepfake Detection via Token-Level Shuffling and Mixing (AAAI 2025 · similarity 0.85)
- Towards More General Video-based Deepfake Detection through Facial Component Guided Adaptation for Foundation Model (CVPR 2025 · similarity 0.85)
- From Specificity to Generality: Revisiting Generalizable Artifacts in Detecting Face Deepfakes (NeurIPS 2025 · similarity 0.85)
- D^3: Scaling Up Deepfake Detection by Learning from Discrepancy (CVPR 2025 · similarity 0.85)
- Critical Forgetting-Based Multi-Scale Disentanglement for Deepfake Detection (AAAI 2025 · similarity 0.85)
- DF40: Toward Next-Generation Deepfake Detection (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.