Research map / Video / Video generation
Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a frozen lip-reading expert loss to the local-audio TalkLip (ablation) | TalkLip (l) · LRW | ACC (%): 48.58 -> 79.78 | Helped |
| Add a frozen lip-reading expert loss to the local-audio TalkLip (ablation) | TalkLip (l) · LRS2 | WER₁ (%): 103.57 -> 45.74 | Helped |
| Add contrastive learning to local audio embeddings (ablation) | TalkLip (l) · LRW | LSE-C: 6.44 -> 6.51 | Helped |
| Add contrastive learning to local audio embeddings (ablation) | TalkLip (l) · LRS2 | WER₁ (%): 45.74 -> 38.00 | Helped |
| Replace local audio embeddings with global transformer audio embeddings (ablation) | TalkLip without contrastive learning · LRS2 | WER₁ (%): 45.74 -> 25.31 | Helped |
| Replace local audio embeddings with global transformer audio embeddings (ablation) | TalkLip without contrastive learning · LRW | ACC (%): 79.78 -> 86.57 | Helped |
| Add contrastive learning to global audio embeddings (ablation) | TalkLip (g) · LRS2 | WER₁ (%): 25.31 -> 23.43 | Helped |
| Add contrastive learning to global audio embeddings (ablation) | TalkLip (g) · LRW | LSE-C: 7.01 -> 7.28 | Helped |
| Remove lip-reading loss (ablation) | TalkLip (l) · LRS2 | WER₁ (%): 45.74 -> 103.57 | Hurt |
| Remove both lip-reading and GAN losses (ablation) | TalkLip (l) · LRW | LSE-C: 6.44 -> 5.03 | Hurt |
| Use lip-reading expert supervision, global audio embeddings, and contrastive learning | TalkLip · LRS2 | WER₁ (%): 23.43 vs 82.06 | Helped |
| Use lip-reading expert supervision, global audio embeddings, and contrastive learning | TalkLip · LRW | ACC (%): 87.81 vs 59.98 | Helped |
Closest papers by meaning
- SyncTalkFace: Talking Face Generation with Precise Lip-Syncing via Audio-Lip Memory (AAAI 2022 · similarity 0.93)
- Let There Be Sound: Reconstructing High Quality Speech from Silent Videos (AAAI 2024 · similarity 0.90)
- Perceptually Accurate 3D Talking Head Generation: New Definitions, Speech-Mesh Representation, and Evaluation Metrics (CVPR 2025 · similarity 0.90)
- LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading (ICLR 2024 · similarity 0.89)
- Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language (AAAI 2025 · similarity 0.89)
- PointTalk: Audio-Driven Dynamic Lip Point Cloud for 3D Gaussian-based Talking Head Synthesis (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.