Research map / Vision-language and multimodal / Cross-modal learning
VisualVoice: Audio-Visual Speech Separation With Cross-Modal Consistency
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add facial-attribute and cross-modal cues to lip-motion separation (ablation) | VisualVoice · VoxCeleb2, reliable lip motion | SDR: 9.95 -> 10.2 | Helped |
| add facial-attribute and cross-modal cues to lip-motion separation (ablation) | VisualVoice · VoxCeleb2, unreliable lip motion | SDR: 7.57 -> 8.53 | Helped |
| use lip motion only instead of both visual cues (ablation) | VisualVoice · VoxCeleb2, unreliable lip motion | SDR: 8.53 -> 7.57 | Hurt |
| use static face only instead of both visual cues (ablation) | VisualVoice · VoxCeleb2, reliable lip motion | SDR: 10.2 -> 7.21 | Hurt |
| use static face only instead of both visual cues (ablation) | VisualVoice · VoxCeleb2, unreliable lip motion | SDR: 8.53 -> 7.21 | Hurt |
| add facial-attribute and cross-modal cues to lip-motion separation (ablation) | VisualVoice · VoxCeleb2 with AudioSet noise | SDR: 6.31 -> 6.55 | Helped |
| jointly train cross-modal embeddings with speech separation (ablation) | VisualVoice · VoxCeleb1, seen-heard | AUC: 75.0 -> 84.9 | Helped |
| jointly train cross-modal embeddings with speech separation (ablation) | VisualVoice · VoxCeleb1, unseen-unheard | AUC: 72.4 -> 74.2 | Helped |
Closest papers by meaning
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.89)
- From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech (CVPR 2025 · similarity 0.89)
- ES3: Evolving Self-Supervised Learning of Robust Audio-Visual Speech Representations (CVPR 2024 · similarity 0.87)
- ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.87)
- Continual Audio-Visual Sound Separation (NeurIPS 2024 · similarity 0.87)
- Visual Hallucination Elevates Speech Recognition (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.