Research map / 3D vision / Face and shape
Expression-Preserving Face Frontalization Improves Visually Assisted Speech Processing
Zhiqi Kang, Mostafa Sadeghi, Radu Horaud, Xavier Alameda-Pineda
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dynamic Kalman-style shape update to robust frontalization | RFF · OuluVS2 frontalization | ZNCC: 0.831 -> 0.839 | Improved |
| add robust RFF frontalization to the lip input (ablation) | Res-AV-CVAE · MEAD AVSE SNR 0 dB | STOI: 0.67 -> 0.71 | Improved |
| add dynamic frontalization (DFF) to the lip input (ablation) | Res-AV-CVAE · MEAD AVSE SNR 0 dB | STOI: 0.67 -> 0.73 | Improved |
| add dynamic frontalization (DFF) to the lip input (ablation) | Res-AV-CVAE · MEAD AVSE SNR 0 dB | PESQ: 1.99 -> 2.20 | Improved |
| add dynamic frontalization (DFF) to the lip input (ablation) | Res-AV-CVAE · MEAD AVSE SNR 0 dB | SI-SDR (dB): 4.68 -> 5.87 | Improved |
| use DFF instead of RFF | Res-AV-CVAE · MEAD AVSE SNR 0 dB | SI-SDR (dB): 5.24 -> 5.87 | Improved |
| use ST-GAN frontalization instead of none | Res-AV-CVAE · MEAD AVSE SNR 0 dB | STOI: 0.67 -> 0.68 | No clear effect |
| use DA-GAN frontalization instead of none | Res-AV-CVAE · MEAD AVSE SNR 0 dB | PESQ: 1.99 -> 1.87 | Hurt |
| replace fully connected visual encoder with pretrained ResNet lip-reading backbone | AV-CVAE · MEAD AVSE | STOI/PESQ | Improved |
| fine-tune lip reader with proposed frontalization instead of GAN frontalization | TCN lip-reading model · LRW 500-IWR | WCS (%): 84 -> 88 | Improved |
| fine-tune lip reader with GAN frontalization | TCN lip-reading model · LRW 500-IWR | WCS (%): 87 -> 84 | Hurt |
| fine-tune lip reader with proposed frontalization | TCN lip-reading model · LRW 500-IWR | WCS (%): 87 -> 88 | Improved |
Closest papers by meaning
- Learning Adaptive Spatial Coherent Correlations for Speech-Preserving Facial Expression Manipulation (CVPR 2024 · similarity 0.86)
- DEEPTalk: Dynamic Emotion Embedding for Probabilistic Speech-Driven 3D Face Animation (AAAI 2025 · similarity 0.86)
- Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation (CVPR 2025 · similarity 0.86)
- Faces that Speak: Jointly Synthesising Talking Face and Speech from Text (CVPR 2024 · similarity 0.85)
- Probabilistic Speech-Driven 3D Facial Motion Synthesis: New Benchmarks Methods and Applications (CVPR 2024 · similarity 0.85)
- FaceTalk: Audio-Driven Motion Diffusion for Neural Parametric Head Models (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
