Research map / Vision-language and multimodal / Cross-modal learning
AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AVFF, a two-stage method that pre-trains audio-visual representations on real talking-face videos (LRS3) via contrastive plus masked-autoencoding with complementary masking and cross-modal fusion, then fine-tunes a classifier on FakeAVCeleb for video deepfake detection.
Problem
Video deepfake detection where audio, visual, or both modalities may be synthetic; existing supervised audio-visual detectors may overfit to training manipulations and miss audio-visual correspondences that generalize to unseen deepfakes.
Approach
Self-supervised audio-visual representation learning on real videos only, combining a bi-directional contrastive loss with an MAE-style autoencoder in which complementary temporal slice masking forces each modality's masked slices to be predicted from the other modality via A2V/V2A networks, followed by supervised real/fake classification on the learned features.
Key results
- Intra-dataset on FakeAVCeleb (70/30 split): 98.6% accuracy and 99.1% AUC, vs. AVoiD-DF's 83.7% ACC and 89.2% AUC.
- Cross-manipulation on FakeAVCeleb: AVG-FV AUC 99.5 and AP 98.5 averaged across fake-visual categories, outperforming baselines such as RealForensics (AUC 97.1).
- Cross-dataset on KoDF (trained on FakeAVCeleb): AP 93.1 and AUC 95.5, behind RealForensics on AP (95.7) and AUC (93.6) only in the AP column; AVFF is best on AUC among the listed methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AVFF instead of AVoiD-DF | FakeAVCeleb intra-dataset detection · FakeAVCeleb intra-dataset | ACC: 83.7 -> 98.6 | Helped |
| Use AVFF instead of AVoiD-DF | FakeAVCeleb intra-dataset detection · FakeAVCeleb intra-dataset | AUC: 89.2 -> 99.1 | Helped |
| Use AVFF instead of RealForensics | FakeAVCeleb cross-manipulation detection · FakeAVCeleb cross-manipulation AVG-FV | AUC: 97.1 -> 99.5 | Helped |
| Use AVFF instead of RealForensics | FakeAVCeleb cross-manipulation detection · FakeAVCeleb cross-manipulation AVG-FV | AP: 95.3 -> 98.5 | Helped |
| Add the autoencoding objective, complementary masking, cross-modal fusion, and decoding (ablation) | AVFF · FakeAVCeleb | AP: 84.2 -> 96.7 | Helped |
| Add A2V/V2A cross-modal fusion instead of shared learnable masked tokens (ablation) | AVFF · FakeAVCeleb | AP: 87.2 -> 96.7 | Helped |
| Use complementary masking instead of random masking (ablation) | AVFF · FakeAVCeleb | AP: 78.9 -> 96.7 | Helped |
| Concatenate feature and cross-modal embeddings rather than use feature embeddings alone (ablation) | AVFF · FakeAVCeleb | AP: 89.7 -> 96.7 | Helped |
| Concatenate feature and cross-modal embeddings rather than use cross-modal embeddings alone (ablation) | AVFF · FakeAVCeleb | AP: 94.6 -> 96.7 | Helped |
| Use uni-modal patch reduction networks instead of mean pooling (ablation) | AVFF · FakeAVCeleb | AP: 96.5 -> 96.7 | No clear effect |
Surprising findings
- Stage 1 is trained only on real LRS3 videos, yet t-SNE embeddings already separate real and fake FakeAVCeleb samples and different deepfake categories, and AUC reaches about 90% within 1-3 downstream epochs.
Limitations
- Relies on audio-visual coherence, so asynchronous videos, mismatched audio, multiple speakers, or occlusions are challenging.
- Requires both audio and visual modalities; single-modality videos are unsupported.
- Cross-dataset results are at par with RealForensics rather than clearly better.
Experimental setup
Datasets: LRS3 FakeAVCeleb
Benchmarks: FakeAVCeleb KoDF
Metrics: accuracy average precision AUC
Baselines: Xception LipForensics FTCN CViT RealForensics Emotions Don't Lie MDS AVFakeNet VFD AVoiD-DF AV-DFD AVAD
Builds on: CAV-MAE MARLIN Audio-MAE SyncNet
Closest papers by meaning
- Multi-modal Deepfake Detection via Multi-task Audio-Visual Prompt Learning (AAAI 2025 · similarity 0.92)
- Circumventing Shortcuts in Audio-visual Deepfake Detection Datasets with Unsupervised Learning (CVPR 2025 · similarity 0.90)
- Standing on the Shoulders of Giants: Reprogramming Visual-Language Model for General Deepfake Detection (AAAI 2025 · similarity 0.88)
- Generalizing Deepfake Video Detection with Plug-and-Play: Video-Level Blending and Spatiotemporal Adapter Tuning (CVPR 2025 · similarity 0.88)
- Multi-View Collaborative Learning Network for Speech Deepfake Detection (AAAI 2025 · similarity 0.87)
- Adaptive Compression and Targeted Perturbation: A Unified Framework for Generalized Audio Deepfake Detection (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.