Research map / Video / Video segmentation and tracking
SpeechForensics: Audio-Visual Speech Representation Learning for Face Forgery Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns audio-visual speech representations on real videos with a masked-prediction plus frame-wise alignment objective (AV-HuBERT-based), then detects face forgeries without fake training data by measuring audio-visual cosine similarity.
Problem
Face forgery video detection that generalizes to unseen manipulations, datasets, and common perturbations without training on fake videos.
Approach
Uses audio-visual speech representations learned only from real videos (local frame-wise alignment plus global masked prediction) and flags videos whose lip-movement and audio embeddings have low cosine similarity, with fixed-offset sliding-window or DTW alignment to handle audio-visual time offsets.
Key results
- Cross-dataset: 99.0% AUC on FakeAVCeleb and 91.7% on KoDF, averaging 95.4%, outperforming RealForensics by 8.8% and 7.4% respectively.
- Cross-manipulation on FF++ HQ: 97.6% average AUC without any fake training data, comparable to supervised FTCN (98.3%).
- Under perturbations, average AUC 94.1% with a drop of only 3.5%, versus 92.0% (drop 7.6%) for RealForensics.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add global temporal information modeling to local phonetic-level speech representations (ablation) | SpeechForensics · FF++ cross-manipulation Avg | AUC (%): 88.7 -> 97.6 | Helped |
| add global temporal information modeling to local phonetic-level speech representations (ablation) | SpeechForensics · FakeAVCeleb Overall | AUC (%): 19.0 -> 99.0 | Helped |
| extend video clip length up to 16 seconds (ablation) | SpeechForensics · FF++ FaceSwap and Face2Face | AUC | Helped |
| use dynamic time warping rather than fixed time-offset alignment (ablation) | SpeechForensics · FF++ | AUC | Hurt |
| increase the sliding-window length by increasing maximum offset τ (ablation) | SpeechForensics · FF++ | AUC | Helped |
| use a larger AVHuBERT model (ablation) | SpeechForensics · FF++ and FakeAVCeleb | AUC | Helped |
| train on more data, combining LRS3 and VoxCeleb2 rather than LRS3 alone (ablation) | SpeechForensics · FF++ and FakeAVCeleb | AUC | Helped |
| apply unseen perturbations | SpeechForensics · FF++ | AUC (%): 97.6 -> 94.1 | Hurt |
| use SpeechForensics instead of RealForensics | cross-dataset face forgery detection · FakeAVCeleb | AUC (%): 90.2 -> 99.0 | Helped |
| use SpeechForensics instead of RealForensics | cross-dataset face forgery detection · KoDF | AUC (%): 84.3 -> 91.7 | Helped |
Surprising findings
- The local-only SpeechForensics-Local variant fails on Wav2Lip (about 0.1% AUC), showing frame-level lip sync alone is not enough.
- The dynamic time warping alignment performs slightly worse than the fixed-offset sliding window, which the authors attribute to DTW inflating fake-video similarity.
Limitations
- Relies on mouth region; cannot detect forgeries that do not alter the mouth area.
- Performance degrades on extreme samples such as silent clips or audio with heavy ambient noise or background music.
- Lower clean-video AUC than some supervised baselines (97.6% vs 99.6% for LipForensics) on FF++ perturbation tests.
Experimental setup
Datasets: LRS3 VoxCeleb2 FaceForensics++ FakeAVCeleb KoDF
Benchmarks: FaceForensics++ FakeAVCeleb KoDF DeeperForensics perturbations
Metrics: AUC
Baselines: Xception Patch-based Face X-ray LipForensics FTCN RealForensics AVAD SpeechForensics-Local
Builds on: AV-HuBERT AVAD RealForensics LipForensics Dynamic Time Warping
Closest papers by meaning
- Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector (CVPR 2025 · similarity 0.87)
- Identity-Aware Vision-Language Model for Explainable Face Forgery Detection (AAAI 2026 · similarity 0.86)
- Forensics Adapter: Adapting CLIP for Generalizable Face Forgery Detection (CVPR 2025 · similarity 0.85)
- VLForgery Face Triad: Detection, Localization and Attribution via Multimodal Large Language Models (NeurIPS 2025 · similarity 0.85)
- A Hitchhiker's Guide to Fine-Grained Face Forgery Detection Using Common Sense Reasoning (NeurIPS 2024 · similarity 0.85)
- Towards General Visual-Linguistic Face Forgery Detection (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.