Research map / Video / Video segmentation and tracking
Lips Are Lying: Spotting the Temporal Inconsistency between Audio and Visual in Lip-Syncing DeepFakes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LipFD, a dual-headed transformer detector that spots temporal inconsistencies between lip movements and audio spectrograms (plus head/face crops) to identify lip-syncing deepfakes, and releases the AVLips dataset.
Problem
Detecting lip-syncing deepfakes, which keep identity and lack visible artifacts, where existing DeepFake detectors degrade sharply.
Approach
Combine a CLIP ViT-L/14 global encoder over frames and spectrogram with multi-scale crops (lip, face, head) and a learned region-awareness module that weights regions, trained with a region-awareness loss that emphasizes frequently modified lower-face areas.
Key results
- LipFD reaches 95.27% ACC and 93.08% AP on AVLips, versus 91.78% ACC for RealForensics in cross-dataset validation.
- On FF++ and DFDC, LipFD achieves 95.10% and 94.53% ACC respectively.
- Real-world WeChat-style video call simulation yields up to 90.18% accuracy at 100ms latency (streaming media, EN).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LipFD instead of RealForensics | LipSync detection · AVLips | ACC: 91.78 -> 95.27 | Helped |
| use LipFD instead of LipForensics | LipSync detection · FF++ | AP: 93.25 -> 76.98 | Hurt |
| use LipFD instead of RealForensics | LipSync detection · DFDC | AP: 91.62 -> 78.61 | Hurt |
| evaluate on MakeItTalk-generated videos rather than Wav2Lip-generated videos | LipFD · Cross-manipulation generalisation | ACC: 95.27 -> 96.93 | Helped |
| evaluate on TalkLip-generated videos rather than Wav2Lip-generated videos | LipFD · Cross-manipulation generalisation | ACC: 95.27 -> 79.33 | Hurt |
| remove Global Encoder (ablation) | LipFD · LipSync detection | ACC: 95.27 -> 95.07 | Hurt |
| remove Global-Region Encoder (ablation) | LipFD · LipSync detection | ACC: 95.27 -> 72.52 | Hurt |
| remove Region Awareness (ablation) | LipFD · LipSync detection | ACC: 95.27 -> 76.45 | Hurt |
| replace CLIP:ViT/L14 with CLIP:ViT/B16 (ablation) | LipFD · LipSync detection | ACC: 95.27 -> 95.00 | Hurt |
| replace CLIP:ViT/L14 with ImageNet:ViT/L16 (ablation) | LipFD · LipSync detection | ACC: 95.27 -> 93.28 | Hurt |
| replace CLIP:ViT/L14 with ImageNet:ViT/B16 (ablation) | LipFD · LipSync detection | ACC: 95.27 -> 93.27 | Hurt |
| replace CLIP:ViT/L14 with ImageNet:Swin-B (ablation) | LipFD · LipSync detection | ACC: 95.27 -> 94.66 | Hurt |
Surprising findings
- On the FF++ and DFDC datasets, LipFD has lower AP than LipForensics and RealForensics, despite higher accuracy.
- Detection accuracy drops to 52.89% at 500ms latency for streaming media and Chinese videos score much lower than English.
Limitations
- Performance drops considerably on non-English videos (Chinese accuracy much lower).
- Accuracy degrades under latency and network delay that desynchronizes audio and video.
- Lower AP than prior methods on general face-swap datasets (FF++, DFDC) because it targets subtle lip edits.
Experimental setup
Datasets: LRS3 AVLips FF++ DFDC
Benchmarks: AVLips FF++ DFDC
Metrics: ACC AP FPR FNR AUC
Baselines: CViT DoubleStream UniversalFakeDetect SelfBlendedImages RealForensics LipForensics
Builds on: CLIP RealForensics LipForensics
Closest papers by meaning
- OmniSync: Towards Universal Lip Synchronization via Diffusion Transformers (NeurIPS 2025 · similarity 0.88)
- Generalizing Deepfake Video Detection with Plug-and-Play: Video-Level Blending and Spatiotemporal Adapter Tuning (CVPR 2025 · similarity 0.86)
- Towards a Universal Synthetic Video Detector: From Face or Background Manipulations to Fully AI-Generated Content (CVPR 2025 · similarity 0.86)
- AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection (CVPR 2024 · similarity 0.86)
- DF40: Toward Next-Generation Deepfake Detection (NeurIPS 2024 · similarity 0.86)
- Circumventing Shortcuts in Audio-visual Deepfake Detection Datasets with Unsupervised Learning (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.