Research map / 3D vision / 3D generation
Lipper: Synthesizing Thy Speech Using Multi-View Lipreading
Yaman Kumar, Rohit Jain, Khwaja Mohd. Salik, Rajiv Ratn Shah, Yifang Yin, Roger Zimmermann
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use multi-view input rather than one view, with the best tested combination 0°+45°+60° (ablation) | Lipper · OuluVS2 speaker-dependent | PESQ: 2.002 -> 2.315 | Improved |
| use the 0°+45°+60° combination instead of all five views (ablation) | Lipper · OuluVS2 speaker-dependent | PESQ: 2.086 -> 2.315 | Improved |
| add 90° to the best 0°+45°+60° view combination (ablation) | Lipper · OuluVS2 speaker-dependent | PESQ: 2.315 -> 2.147 | Hurt |
| use 0°+45°+60° rather than the 0°+45° pair (ablation) | Lipper · OuluVS2 speaker-dependent | PESQ: 2.130 -> 2.315 | Improved |
| add a second view to single-view reconstruction (ablation) | Lipper · OuluVS2 speaker-dependent, tested two-view combinations | PESQ: Single-view 0°: 2.002; 0°+45°: 2.130 | Improved |
| vary which views are combined (ablation) | Lipper · OuluVS2 speaker-dependent, three-view combinations | PESQ: 1.814 to 2.315 | Mixed |
| vary which views are combined (ablation) | Lipper · OuluVS2 speaker-dependent, four-view combinations | PESQ: 1.916 to 2.147 | Mixed |
| use Lipper instead of the prior speech-reconstruction system | Lipper · OuluVS2 speaker-dependent, single views | PESQ: Lipper: 2.002, 1.750, 1.642, 1.744, 1.804; prior: 1.72, 1.57, 1.48, 1.46, 1.52 | Improved |
| test on held-out phrases rather than phrases seen during training (ablation) | Lipper · OuluVS2 speaker-dependent OOV | PESQ: Phrase scores: 1.79, 1.66, 1.82, 1.84, 1.57, 1.64, 1.68, 1.55, 1.46, 1.60 | Hurt |
| evaluate on unseen speakers rather than speaker-dependent data (ablation) | Lipper · OuluVS2 speaker-independent | PESQ | Hurt |
| train text prediction with a random across-speaker split rather than a speaker-held-out split (ablation) | Lipper text-prediction model · OuluVS2 text prediction | accuracy: 78.5 -> 97.0 | Improved |
| generate audio from lip motion in fixed short frame sequences instead of waiting for a full phrase and classifying it | Lipper · OuluVS2 delay comparison | delay: Lipper: 0.169; prior system: 0.94 to 1.95 | Improved |
Closest papers by meaning
- LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading (ICLR 2024 · similarity 0.91)
- Sub-Word Level Lip Reading With Visual Attention (CVPR 2022 · similarity 0.89)
- Let There Be Sound: Reconstructing High Quality Speech from Silent Videos (AAAI 2024 · similarity 0.89)
- Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language (AAAI 2025 · similarity 0.87)
- Language Without Borders: A Dataset and Benchmark for Code-Switching Lip Reading (NeurIPS 2024 · similarity 0.87)
- Restoring Speaking Lips from Occlusion for Audio-Visual Speech Recognition (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
