Research map / Video / Video segmentation and tracking
Circumventing Shortcuts in Audio-visual Deepfake Detection Datasets with Unsupervised Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reveals a leading-silence shortcut in FakeAVCeleb and AV-Deepfake1M and proposes AVH-Align, an unsupervised audio-video misalignment detector trained only on real videos on top of AV-HuBERT features.
Problem
Audio-visual deepfake detection datasets contain spurious artifacts (a leading silence in fake videos) that supervised detectors exploit, inflating reported performance and hurting generalization.
Approach
Shift from supervised to unsupervised learning by training only on real videos: an MLP alignment network over frozen self-supervised AV-HuBERT audio and visual frame features is trained with a InfoNCE-style loss to match each video frame to its audio frame, and fakes are flagged by misalignment scores.
Key results
- AVH-Align is essentially unaffected by trimming the leading silence on FakeAVCeleb (94.6 AUC both) and changes only 2.4 AUC on AV-Deepfake1M (AUC 85.9 to 83.5).
- The supervised variant AVH-Align/sup drops 16.9 AUC on AV-Deepfake1M when trained on FakeAVCeleb-free AV-Deepfake1M setup after trimming (training on FakeAVCeleb: 85.9 to 69.0 cross-set trimmed drop reported), showing reliance on silence.
- On the official AV-Deepfake1M test set, AVH-Align reaches 85.24% AUC, while a simple silence classifier reaches 98.44% AUC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use leading silence duration as the classifier feature | FakeAVCeleb and AV-Deepfake1M · RVRA-FVFA test subsets | AUC: 98.4% and 98.2% | Helped |
| Trim the leading silence | Silence classifier · FakeAVCeleb and AV-Deepfake1M | AUC: 98.4 -> 54.8; 98.2 -> 50.6 | Hurt |
| Trim the leading silence | RawNet2 · FakeAVCeleb and AV-Deepfake1M | AUC: 99.9 -> 97.3; 99.9 -> 88.1 | Hurt |
| Trim the leading silence | MDS · FakeAVCeleb and AV-Deepfake1M | AUC: 90.4 -> 73.8; 99.2 -> 54.9 | Hurt |
| Trim the leading silence | AVAD · FakeAVCeleb and AV-Deepfake1M | AUC: 95.2 -> 95.2; 52.9 -> 52.9 | No clear effect |
| Train the alignment network on real data with AV-HuBERT features | AVH-Align · AV-Deepfake1M | AUC: 85.9 vs 68.8 | Helped |
| Trim the leading silence | AVH-Align · FakeAVCeleb and AV-Deepfake1M | AUC: 94.6 -> 94.6; 85.9 -> 83.5 | No clear effect |
| Train with fake samples and a supervised classification loss | AVH-Align/sup · AV-Deepfake1M trimmed | AUC: 100.0 -> 83.1 | Hurt |
| Remove feature normalization (ablation) | AVH-Align · FakeAVCeleb and AV-Deepfake1M | AUC: 94.6 -> 92.4; 85.9 -> 84.8 | Hurt |
| Reduce training set to 8.7k samples (ablation) | AVH-Align · FakeAVCeleb and AV-Deepfake1M | AUC: 94.6 -> 89.1; 85.9 -> 82.5 | Hurt |
| Use mean score pooling instead of log-sum-exp (ablation) | AVH-Align · FakeAVCeleb and AV-Deepfake1M | AUC: 94.6 -> 97.4; 85.9 -> 76.8 | Mixed |
| Use AV-HuBERT pretrained on LRS3 only (ablation) | AVH-Align · FakeAVCeleb and AV-Deepfake1M | AUC: 94.6 -> 84.8; 85.9 -> 60.3 | Hurt |
Surprising findings
- A simple leading-silence classifier reaches over 98% AUC on FakeAVCeleb and AV-Deepfake1M, revealing a spurious shortcut.
- Audio-only RawNet2 remains the top performer even after silence trimming, suggesting audio streams are underused.
Limitations
- Unsupervised AVH-Align has modest absolute performance on some settings (e.g., 69% AP-level results on FakeAVCeleb trimmed for supervised variant; AUC 85.24 on official test).
- Cause of the leading silence bias is only speculated, not verified with the generation process.
- Other unrevealed biases may remain undetected.
Experimental setup
Datasets: FakeAVCeleb AV-Deepfake1M LAV-DF AVLips DFDC
Benchmarks: FakeAVCeleb AV-Deepfake1M AVLips AV-Deepfake1M official test set
Metrics: AUC AP
Baselines: RawNet2 MDS AVAD SpeechForensics AVH-Align/sup Meso4 MesoInception4 Xception EfficientViT MARLIN
Builds on: AV-HuBERT AVAD InfoNCE
Closest papers by meaning
- AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection (CVPR 2024 · similarity 0.90)
- Adaptive Compression and Targeted Perturbation: A Unified Framework for Generalized Audio Deepfake Detection (NeurIPS 2026 · similarity 0.88)
- I Can Hear You: Selective Robust Training for Deepfake Audio Detection (ICLR 2025 · similarity 0.88)
- Multi-modal Deepfake Detection via Multi-task Audio-Visual Prompt Learning (AAAI 2025 · similarity 0.87)
- Exposing the Deception: Uncovering More Forgery Clues for Deepfake Detection (AAAI 2024 · similarity 0.86)
- Generalizing Deepfake Video Detection with Plug-and-Play: Video-Level Blending and Spatiotemporal Adapter Tuning (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.