Research map / Video / Video segmentation and tracking
SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 3,652 hours of synthetic videos to the 30-hour real labeled training set (ablation) | SynthVSR Conformer-BASE · LRS3 test, 30-hour labeled setting | WER (%): 104.0 -> 44.7 | Helped |
| Use synthetic training data alongside real labeled videos rather than synthetic data alone (ablation) | SynthVSR Conformer-BASE · LRS3 test, 30-hour labeled setting | WER (%): 100.3 -> 44.7 | Helped |
| Add 3,652 hours of synthetic videos to 438 hours of real labeled LRS3 (ablation) | SynthVSR Conformer-BASE · LRS3 test, 438-hour labeled setting | WER (%): 36.7 -> 28.4 | Helped |
| Add 3,652 hours of synthetic videos to 3,068 hours of labeled and pseudo-labeled training data (ablation) | SynthVSR Conformer-BASE · LRS3 test, high-resource setting | WER (%): 21.2 -> 19.4 | Helped |
| Add both visual and linguistic VSR perceptual losses to lip-animation training (ablation) | SynthVSR, 438-hour LRS3 setting · LRS3 test | WER (%): 32.2 -> 30.8 | Helped |
| Add visual VSR perceptual loss to lip-animation training (ablation) | SynthVSR, 438-hour LRS3 setting · LRS3 test | WER (%): 32.2 -> 31.7 | Helped |
| Add linguistic VSR perceptual loss to lip-animation training (ablation) | SynthVSR, 438-hour LRS3 setting · LRS3 test | WER (%): 32.2 -> 31.4 | Helped |
| Use CelebA rather than LRS3-pretrain as the lip-image source (ablation) | SynthVSR · LRS3 test, 438-hour setting | WER (%): 32.1 -> 30.8 | Helped |
| Scale synthetic speech sources from a single corpus to TED-LIUM 3, Librispeech, and Common Voice combined (ablation) | SynthVSR · LRS3 test, 438-hour labeled setting | WER (%): 30.9, 30.8, and 30.1 -> 28.4 | Helped |
| Double Librispeech synthetic videos by generating two videos per speech clip with two CelebA lip images (ablation) | SynthVSR · LRS3 test, 438-hour labeled setting, no LM | WER (%): 30.8 -> 30.1 | Helped |
| Use TTS-generated speech rather than original Librispeech speech to generate synthetic videos (ablation) | SynthVSR · LRS3 test, 438-hour labeled setting | WER (%): 36.7 -> 32.9 | Helped |
| Increase the VSR model from Conformer-BASE to Conformer-LARGE | SynthVSR with 3,068 hours labeled and 3,652 synthetic hours · LRS3 test, high-resource setting | WER (%): 18.7 -> 16.9 | Helped |
Closest papers by meaning
- LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading (ICLR 2024 · similarity 0.88)
- Restoring Speaking Lips from Occlusion for Audio-Visual Speech Recognition (AAAI 2024 · similarity 0.88)
- VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness (NeurIPS 2026 · similarity 0.88)
- ProsodyTalker: 3D Visual Speech Animation via Prosody Decomposition (AAAI 2025 · similarity 0.87)
- From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech (CVPR 2025 · similarity 0.87)
- Visual Hallucination Elevates Speech Recognition (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.