Research map / Large language models / Language, safety and interpretability
Language Without Borders: A Dataset and Benchmark for Code-Switching Lip Reading
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CSLR, a 71.3-hour Chinese-English code-switching sentence-level lip reading dataset (85,560 videos from 62 speakers) and benchmarks lip reading backbones and adapted code-switching speech recognition methods on it.
Problem
Lip reading research has focused on monolingual data, and code-switching (mixing languages within an utterance) has not been studied for visual speech recognition, leaving no suitable benchmark for bilingual lip reading.
Approach
Collect the first sentence-level code-switching lip reading dataset (Chinese, English, and Chinese-English mixed utterances recorded on smartphone front cameras), and adapt code-switching speech recognition methods (Bi-Encoder CTC, Language Aware Training, LR-MoE) to lip reading.
Key results
- Conformer back-end gave the best code-switching results among backbones (CER 35.32%, WER 49.05%, MER 40.35%).
- LR-MoE CTC achieved the best code-switching result (CER 33.01%, WER 45.37%, MER 37.54%), versus 40.35% MER for vanilla CTC.
- CSLR is reported as challenging, with unsatisfied accuracy and confusion between Chinese and English lip patterns.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a Bi-Encoder with monolingual pre-training instead of a single-stream Vanilla CTC model | ShuffleNet+Conformer · CSLR code-switching | CER: 35.32% -> 33.81% | Helped |
| Add Language Aware Training to the Bi-Encoder (ablation) | ShuffleNet+Conformer Bi-Encoder · CSLR code-switching | CER: 33.81% -> 33.54% | Helped |
| Add Language Aware Training to the Bi-Encoder (ablation) | ShuffleNet+Conformer Bi-Encoder · CSLR code-switching | WER: 46.33% -> 46.08% | Helped |
| Add Language Aware Training to the Bi-Encoder (ablation) | ShuffleNet+Conformer Bi-Encoder · CSLR code-switching | MER: 38.40% -> 38.13% | Helped |
| Use Language-Routing Mixture-of-Experts with LiD loss instead of Bi-Encoder+LAT | ShuffleNet+Conformer · CSLR code-switching | CER: 33.54% -> 33.01% | Helped |
| Use Language-Routing Mixture-of-Experts with LiD loss instead of Bi-Encoder+LAT | ShuffleNet+Conformer · CSLR code-switching | WER: 46.08% -> 45.37% | Helped |
| Use Language-Routing Mixture-of-Experts with LiD loss instead of Bi-Encoder+LAT | ShuffleNet+Conformer · CSLR code-switching | MER: 38.13% -> 37.54% | Helped |
| Use Conformer instead of BiGRU as the back-end | ShuffleNet · CSLR code-switching | CER: 36.91% -> 35.32% | Helped |
Surprising findings
- Monolingual-trained models do not transfer well to code-switching; code-switching error rates remain high even for best methods.
Limitations
- Privacy concerns: identifiable facial videos cannot be fully anonymized without losing utility.
- Visual ambiguity between phonemes (e.g., p/b) makes it hard to infer language from lip movements.
- Dataset limited to Chinese-English bilingual speech and 62 speakers.
Experimental setup
Datasets: CSLR
Benchmarks: CSLR
Metrics: CER WER MER
Baselines: Vallina CTC Bi-Encoder CTC Bi-Encoder+LAT LR-MoE CTC MSTCN BiGRU
Builds on: Conformer Bi-Encoder CTC Language Aware Training LR-MoE
Closest papers by meaning
- SwitchLingua: The First Large-Scale Multilingual and Multi-Ethnic Code-Switching Dataset (NeurIPS 2025 · similarity 0.90)
- LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading (ICLR 2024 · similarity 0.88)
- Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language (AAAI 2025 · similarity 0.88)
- Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · similarity 0.85)
- Let There Be Sound: Reconstructing High Quality Speech from Silent Videos (AAAI 2024 · similarity 0.85)
- MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.