Research map / Vision-language and multimodal / Cross-modal learning
Not Only Vision: Evolve Visual Speech Recognition via Peripheral Information
Zhaoxin Yuan, Shuang Yang, Shiguang Shan, Xilin Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add all peripheral information (guidance, expertise, perturbation) | VSR with Llama3.1-8B · LRS3 | WER: 24.92 -> 22.03 | Improved |
| add contextual guidance (title and description) (ablation) | Synergy LoRA · LRS3 | WER: 25.03 -> 23.46 | Improved |
| add all contextual guidance types incl. speaker description (ablation) | Synergy LoRA · LRS3 | WER: 25.03 -> 22.99 | Improved |
| add task expertise (Type B) without guidance (ablation) | Synergy LoRA · LRS3 | WER: 25.03 -> 24.44 | Improved |
| add task expertise (Type B) on top of guidance (ablation) | Synergy LoRA · LRS3 | WER: 23.46 -> 23.09 | Improved |
| add task-aware prompt to task expertise (ablation) | Synergy LoRA Type A · LRS3 | WER: 24.83 -> 24.62 | Improved |
| add output constraint to task expertise (ablation) | Synergy LoRA Type A · LRS3 | WER: 24.91 -> 24.62 | Improved |
| add random-token linguistic perturbation (ablation) | Synergy LoRA · LRS3 | WER: 25.03 -> 24.62 | Improved |
| add natural text clip perturbation (ablation) | Synergy LoRA · LRS3 | WER: 25.03 -> 24.65 | Improved |
| add random perturbation to guidance training (ablation) | Synergy LoRA · LRS3 | WER: 23.46 -> 23.12 | Improved |
| use Synergy LoRA instead of vanilla LoRA with perturbed guidance (ablation) | LoRA · LRS3 | WER: 24.05 -> 23.12 | Improved |
| use Synergy LoRA design (e) instead of vanilla LoRA (a) (ablation) | LoRA · LRS3 with peripheral information | WER: 23.92 -> 22.03 | Improved |
Closest papers by meaning
- VSRo-200: A Romanian Visual Speech Recognition Dataset for Studying Supervision and Multimodal Robustness (NeurIPS 2026 · similarity 0.86)
- Visual Hallucination Elevates Speech Recognition (AAAI 2024 · similarity 0.85)
- Unified Speech Recognition: A Single Model for Auditory, Visual, and Audiovisual Inputs (NeurIPS 2024 · similarity 0.85)
- Learning Visual Speech Representations via Cross-Modal Distillation and Joint Face-Lip Modeling (NeurIPS 2026 · similarity 0.85)
- Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow (CVPR 2026 · similarity 0.85)
- From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
