Research map / Label-efficient and robust learning / Recognition and long-tail
Exploring Unbiased Deepfake Detection via Token-Level Shuffling and Mixing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UDD, a plug-and-play ViT-based deepfake detector that identifies position and content biases and counters them with token-level shuffling and mixing branches plus feature-level contrastive and logit-level JS alignment losses, trained with LoRA on FF++ (c23).
Problem
Deepfake detectors generalize poorly across datasets even when forgery methods are identical, because they overfit to forgery-irrelevant factors: spatial position (e.g., image center) and content (background, hair, identity).
Approach
Intervene on position and content biases inside a ViT by shuffling blockwise patch tokens with randomly resampled position embeddings (position branch) and mixing tokens from same-label samples at intermediate layers (content branch), then align features and logits across the original and intervened branches.
Key results
- Trained on FF++ (c23), video-level AUC reaches 93.1% on CDF and 81.2% on DFDC, state of the art in the paper's comparison.
- Ablation on CDF shows the baseline ViT-B (CLIP) at 88.41% video-level AUC rising to 93.13% with the full method.
- Center-region cutout of 48x48 gives 97.40% frame AUC for the method versus 93.43% for the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add token shuffling and token mixing branches (ablation) | ViT-B (CLIP) deepfake detector · CDF | video-level AUC (%): 90.70% to 93.13% | Helped |
| Add token shuffling and token mixing branches (ablation) | ViT-B (CLIP) deepfake detector · DFDC | video-level AUC (%): 78.25% to 81.21% | Helped |
| Add the shuffling branch (ablation) | UDD · CDF | video-level AUC (%): 91.25 to 93.13 | Helped |
| Add the mixing branch (ablation) | UDD · CDF | video-level AUC (%): 90.51 to 93.13 | Helped |
| Add the contrastive feature loss (ablation) | UDD · CDF | video-level AUC (%): 91.20 to 93.13 | Helped |
| Add the logit alignment loss (ablation) | UDD · CDF | video-level AUC (%): 90.52 to 93.13 | Helped |
| Use CLIP rather than IN21k pretrained weights (ablation) | UDD · CDF | video-level AUC (%): 88.56 to 93.13 | Helped |
| Insert mixing in the mid stage rather than before the first block (ablation) | UDD mixing branch · Average across CDF, DFDC, and DFD | video-level AUC (%): 87.17% to 89.07% | Helped |
| Set contrastive-loss weight λ1 to 0.1 rather than 0 (ablation) | UDD · CDF | frame-level AUC (%): 85.59% to 86.74% | Helped |
| Increase contrastive-loss weight λ1 from 0.1 to 1 (ablation) | UDD · CDF | frame-level AUC (%): 86.74 to 83.55 | Hurt |
| Set alignment-loss weight λ2 to 0.1 rather than 0 (ablation) | UDD · CDF | frame-level AUC (%): 85.75 to 86.74 | Helped |
| Set mixing ratio γ to 0.3 rather than 0.7 (ablation) | UDD mixing branch · CDF | frame-level AUC (%): 83.93% to 86.74% | Helped |
Surprising findings
- Generalization gaps persist even when the same forgery method is used, attributed to position and content bias rather than only forgery-method differences.
Limitations
- Gains on some datasets are modest (e.g., DFDC ablation of pretrained IN21k weights improves only slightly).
- Mixing and alignment assume same-label mixing and that forgery features are local; larger mixing ratios degrade performance.
Experimental setup
Datasets: FaceForensics++ (c23)
Benchmarks: Celeb-DF DFDC-Preview DFDC DFD FaceForensics++ (c23) robustness WildDeepfake FaceShifter FFIW
Metrics: AUC (frame-level) AUC (video-level)
Baselines: Xception Efficient-b4 RECCE UCF IID ICT LSDA TALL++ UIA-ViT SBIs ViT-B (CLIP) ViT-B (IN21k)
Builds on: LoRA SimCLR vision transformer DeepfakeBench
Closest papers by meaning
- FreqDebias: Towards Generalizable Deepfake Detection via Consistency-Driven Frequency Debiasing (CVPR 2025 · similarity 0.90)
- Transcending Forgery Specificity with Latent Space Augmentation for Generalizable Deepfake Detection (CVPR 2024 · similarity 0.90)
- Exposing the Deception: Uncovering More Forgery Clues for Deepfake Detection (AAAI 2024 · similarity 0.90)
- Generalizing Deepfake Video Detection with Plug-and-Play: Video-Level Blending and Spatiotemporal Adapter Tuning (CVPR 2025 · similarity 0.88)
- DF40: Toward Next-Generation Deepfake Detection (NeurIPS 2024 · similarity 0.88)
- Can We Leave Deepfake Data Behind in Training Deepfake Detector? (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.