Research map / Video / Video segmentation and tracking
Spatio-Temporal Fusion Based Convolutional Sequence Learning for Lip Reading
Xingxuan Zhang, Feng Cheng, Shilin Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add short-range temporal fusion (TF-block-a) instead of position-wise feed-forward layers (ablation) | Conv-seq2seq · LRW | WER (%): 22.0 -> 19.3 | Improved |
| add multi-scale branches (kernel 1 and shortcut) to TF-block-a, giving TF-block-b (ablation) | Conv-seq2seq · LRW | WER (%): 19.3 -> 18.7 | Improved |
| add multi-scale branches (TF-block-b vs TF-block-a) (ablation) | Conv-seq2seq · LRS2-BBC | WER (%): 58.5 -> 55.6 | Improved |
| use causal convolution in the encoder (TF-block-c instead of b) (ablation) | Conv-seq2seq · LRW | WER (%): 18.7 -> 20.1 | Hurt |
| add bidirectional mechanism to causal encoder (TF-block-d vs c) (ablation) | Conv-seq2seq · LRW | WER (%): 20.1 -> 18.8 | Improved |
| use Conv-seq2seq with TF-block-b instead of Transformer-seq2seq | sequence model · LRS3-TED | WER (%): 70.8 -> 65.5 | Improved |
| replace global average pooling by simplified STFM (ablation) | Conv-seq2seq · LRW | WER (%): 18.7 -> 16.8 | Improved |
| add temporal convolutions to STFM-simple (full STFM) (ablation) | Conv-seq2seq · LRS3-TED | WER (%): 62.5 -> 60.1 | Improved |
| replace global average pooling by STFM (ablation) | Conv-seq2seq · LRS2-BBC | WER (%): 55.6 -> 51.7 | Improved |
| replace global average pooling by STFM (ablation) | TM-CTC · LRW | WER (%): 35.2 -> 33.7 | Improved |
| replace global average pooling by STFM (ablation) | TM-seq2seq · LRS3-TED | WER (%): 69.8 -> 65.0 | Improved |
| restrict self-attention to local width 50 at test time (ablation) | Conv-seq2seq · LRW | WER (%): 0.8% drop | No clear effect |
Closest papers by meaning
- Sub-Word Level Lip Reading With Visual Attention (CVPR 2022 · similarity 0.88)
- MTGA: Multi-View Temporal Granularity Aligned Aggregation for Event-Based Lip-Reading (AAAI 2025 · similarity 0.88)
- Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language (AAAI 2025 · similarity 0.87)
- Language Without Borders: A Dataset and Benchmark for Code-Switching Lip Reading (NeurIPS 2024 · similarity 0.84)
- LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading (ICLR 2024 · similarity 0.84)
- Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
