Research map / Vision-language and multimodal / Cross-modal learning
MUTE: Multi-Level Alignment Uncoupling Against Talking-Head Exploitation for Voice Protection
Donghyun Kim, Jin Hong, Seungmin Kim, Dain Kim, Junseok Kwon, Daeseon Choi
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MUTE, an audio protection framework that perturbs speech to disrupt talking-head generation models' audio-visual alignment and prevent misuse in deepfakes.
Problem
Talking-head generation models synthesize realistic facial animation from audio and are vulnerable to misuse in multimodal deepfakes; effective protection of the audio input remains underexplored.
Approach
Uses multi-level alignment uncoupling: an audio protection perturbation is designed to break the audio-visual alignment that talking-head models rely on, so generated faces no longer follow the protected audio.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MUTE instead of strongest audio baseline CAF | audio protection · TalkingHead-1KH | SyncNet Sync: 2.173 -> 1.332 | Improved |
| use MUTE instead of CAF | audio protection · black-box SadTalker | SyncNet Sync: 5.425 -> 4.703 | Improved |
| add frame-wise embedding cosine loss instead of global (ablation) | MUTE representation-level loss · TalkingHead-1KH (Hallo) | SyncNet Sync: 2.484 -> 1.332 | Improved |
| add STFT loss (lambda=5) to MUTE (ablation) | MUTE · clean and down/upsampled audio | SyncNet Sync: 1.492 -> 1.332 (clean); 3.178 -> 2.914 (Down/Up) | Improved |
| use second-order term in sensitivity score instead of first-order only (ablation) | target attention · TalkingHead-1KH (Hallo) | SyncNet Sync: 1.383 -> 1.332 | Improved |
| use two-phase alignment-then-representation optimization (rho=0.1) vs single objective (ablation) | MUTE · Hallo, Audio2Head, SadTalker | SyncNet Sync: rho=1.0: 3.920/3.958/5.048; rho=0.0: 1.427/3.904/5.063; rho=0.1: 1.332/3.627/4.703 | Improved |
| increase perturbation budget epsilon from 0.004 to 0.010 (ablation) | MUTE · TalkingHead-1KH | SyncNet Sync and audio quality: Sync 1.332 -> 1.027; WER 11.68 -> 13.30; PESQ 2.522 -> 1.945 | Mixed |
| add speaker-level loss lambda_spk=0.01 (ablation) | MUTE · TTS resynthesis (CosyVoice, XTTS-v2, Chatterbox) | defense success rate: DSR 14.67 -> 67.73 (CosyVoice); Sync 1.332 -> 1.496 | Improved |
| combine MUTE with image-based protection Silencer | Silencer · image+audio protection | SyncNet Sync: 2.170 -> 0.760 | Improved |
| apply adaptive purification attack | MUTE · adaptive attack | SyncNet Sync: 1.332 -> 4.269 | Hurt |
Closest papers by meaning
- Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.89)
- Native Audio-Visual Alignment for Generation (NeurIPS 2026 · similarity 0.88)
- Silence is Golden: Leveraging Adversarial Examples to Nullify Audio Control in LDM-based Talking-Head Generation (CVPR 2025 · similarity 0.86)
- ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.86)
- RVCBench: Benchmarking Robustness of Voice Cloning Across Modern Audio Generation Models (NeurIPS 2026 · similarity 0.86)
- SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
