AI Paper Atlas

Research map / 3D vision / 3D generation

Lipper: Synthesizing Thy Speech Using Multi-View Lipreading

Yaman Kumar, Rohit Jain, Khwaja Mohd. Salik, Rajiv Ratn Shah, Yifang Yin, Roger Zimmermann

AAAI 2019 · 3D vision › 3D generation · approach: 2D-to-3D pretrained representations · 36 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use multi-view input rather than one view, with the best tested combination 0°+45°+60° (ablation)Lipper · OuluVS2 speaker-dependentPESQ: 2.002 -> 2.315Improved
use the 0°+45°+60° combination instead of all five views (ablation)Lipper · OuluVS2 speaker-dependentPESQ: 2.086 -> 2.315Improved
add 90° to the best 0°+45°+60° view combination (ablation)Lipper · OuluVS2 speaker-dependentPESQ: 2.315 -> 2.147Hurt
use 0°+45°+60° rather than the 0°+45° pair (ablation)Lipper · OuluVS2 speaker-dependentPESQ: 2.130 -> 2.315Improved
add a second view to single-view reconstruction (ablation)Lipper · OuluVS2 speaker-dependent, tested two-view combinationsPESQ: Single-view 0°: 2.002; 0°+45°: 2.130Improved
vary which views are combined (ablation)Lipper · OuluVS2 speaker-dependent, three-view combinationsPESQ: 1.814 to 2.315Mixed
vary which views are combined (ablation)Lipper · OuluVS2 speaker-dependent, four-view combinationsPESQ: 1.916 to 2.147Mixed
use Lipper instead of the prior speech-reconstruction systemLipper · OuluVS2 speaker-dependent, single viewsPESQ: Lipper: 2.002, 1.750, 1.642, 1.744, 1.804; prior: 1.72, 1.57, 1.48, 1.46, 1.52Improved
test on held-out phrases rather than phrases seen during training (ablation)Lipper · OuluVS2 speaker-dependent OOVPESQ: Phrase scores: 1.79, 1.66, 1.82, 1.84, 1.57, 1.64, 1.68, 1.55, 1.46, 1.60Hurt
evaluate on unseen speakers rather than speaker-dependent data (ablation)Lipper · OuluVS2 speaker-independentPESQHurt
train text prediction with a random across-speaker split rather than a speaker-held-out split (ablation)Lipper text-prediction model · OuluVS2 text predictionaccuracy: 78.5 -> 97.0Improved
generate audio from lip motion in fixed short frame sequences instead of waiting for a full phrase and classifying itLipper · OuluVS2 delay comparisondelay: Lipper: 0.169; prior system: 0.94 to 1.95Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy