Research map / Vision-language and multimodal / Cross-modal learning
Multi-modal Deepfake Detection via Multi-task Audio-Visual Prompt Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a multi-task audio-visual prompt learning method that adapts frozen CLIP and Whisper encoders with sequential visual prompts and short-time audio prompts, plus a frame-level cross-modal feature matching loss, for multi-modal deepfake video detection.
Problem
Multi-modal deepfake video detection generalizes poorly because existing methods fully fine-tune networks on limited speech-video datasets and use only coarse segment-level audio-visual consistency supervision.
Approach
Freeze visual and audio foundation models (CLIP ViT-B/32 and Whisper-base) and learn sequential visual prompts injected per layer and short-time audio prompts concatenated along the feature dimension, with a two-stream multi-task heads setup, and a frame-level cross-modal feature matching loss that aligns audio and visual features per frame.
Key results
- Intra-dataset on FakeAVCeleb: 99.84% accuracy and 99.98% AUC, matching AVGraph in accuracy and exceeding it in AUC.
- Cross-dataset on KoDF subset: 96.6 AP and 94.5 AUC, outperforming RealForensics by 0.9% in AP and AUC (RealForensics 95.7 AP/93.6 AUC).
- Cross-manipulation on FakeAVCeleb: RVFA AP 97.1 vs AVFF 93.3; the LCMFM loss gives 17.0% accuracy gain on KoDF in ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-task audio-visual prompt learning | FakeAVCeleb detection · FakeAVCeleb | AUC: 99.94 -> 99.98 | No clear effect |
| use multi-task audio-visual prompt learning | KoDF cross-dataset detection · KoDF subset | AP: 93.1 -> 96.6 | Helped |
| use multi-task audio-visual prompt learning | FakeAVCeleb cross-manipulation detection · FakeAVCeleb RVFA | AP: 93.3 -> 97.1 | Helped |
| add the audio classification loss La (ablation) | multi-task audio-visual prompt learning · KoDF | Acc.: 73.5 -> 92.0 | Helped |
| add the audio classification loss La (ablation) | multi-task audio-visual prompt learning · KoDF | AUC: 86.9 -> 94.5 | Helped |
| add the visual classification loss Lv (ablation) | multi-task audio-visual prompt learning · KoDF | Acc.: 77.5 -> 92.0 | Helped |
| add the visual classification loss Lv (ablation) | multi-task audio-visual prompt learning · KoDF | AUC: 90.3 -> 94.5 | Helped |
| use sequential visual prompt insertion instead of VPT prompt replacement (ablation) | multi-task audio-visual prompt learning · KoDF | Acc.: 86.5 -> 92.0 | Helped |
| use sequential visual prompt insertion instead of VPT prompt replacement (ablation) | multi-task audio-visual prompt learning · KoDF | AUC: 90.2 -> 94.5 | Helped |
| use frame-level CMFM loss instead of segment-level loss (ablation) | multi-task audio-visual prompt learning · KoDF | Acc.: 90.0 -> 92.0 | Helped |
| use frame-level CMFM loss instead of segment-level loss (ablation) | multi-task audio-visual prompt learning · KoDF | AUC: 90.6 -> 94.5 | Helped |
Surprising findings
- On the KoDF cross-dataset test, the model reports 96.6 AP but only 94.5 AUC, below AVFF's 95.5 AUC, so the claimed superiority is metric-dependent.
Limitations
- Evaluated only on FakeAVCeleb and a KoDF subset; only CLIP and Whisper base models were tried.
- Frame-level features rely on aligning audio and video at 25fps with a fixed 16-frame segment design.
- Authors note future work on other foundation models and localization, implying current scope is detection only.
Experimental setup
Datasets: FakeAVCeleb KoDF
Benchmarks: FakeAVCeleb KoDF
Metrics: accuracy average precision AUC
Baselines: MDS AV-DFD BA-TFD MMtrace MRDF-CE AVFF AVGraph AVAD Xception LipForensics FTCN RealForensics
Builds on: VPT CLIP Whisper contrastive learning
Closest papers by meaning
- AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection (CVPR 2024 · similarity 0.92)
- Multi-View Collaborative Learning Network for Speech Deepfake Detection (AAAI 2025 · similarity 0.87)
- Circumventing Shortcuts in Audio-visual Deepfake Detection Datasets with Unsupervised Learning (CVPR 2025 · similarity 0.87)
- Standing on the Shoulders of Giants: Reprogramming Visual-Language Model for General Deepfake Detection (AAAI 2025 · similarity 0.86)
- MmAP: Multi-Modal Alignment Prompt for Cross-Domain Multi-Task Learning (AAAI 2024 · similarity 0.85)
- Region-Based Optimization in Continual Learning for Audio Deepfake Detection (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.