Research map / Large language models / Language, safety and interpretability
Cross-Sentence Gloss Consistency for Continuous Sign Language Recognition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cross-Sentence Gloss Consistency (CSGC), which maintains a momentum-updated gloss prototype bank across the dataset with a contrastive loss and an auxiliary cosine-similarity classifier fused with the FC CTC classifier, improving continuous sign language recognition WER on three benchmarks.
Problem
Continuous sign language recognition (CSLR) methods enforce visual-contextual consistency only within individual sentences, missing gloss correlations across sentences in the dataset.
Approach
A global gloss prototype dictionary, updated by momentum from CTC pseudo-labelled gloss features, is used in a contrastive loss to pull same-gloss features together and push others apart; an auxiliary cosine-similarity classifier over the prototypes is fused with the intra-sentence FC classifier at inference.
Key results
- Achieves 18.1% Dev / 19.0% Test WER on PHOENIX14 with ResNet18, versus VAC's 21.2/22.3.
- Achieves 17.2% Dev / 19.5% Test WER on PHOENIX14-T and 27.1% Dev / 26.7% Test WER on CSL-Daily using visual input only.
- Surpasses prior state of the art by 1.6% (PHOENIX14 Dev), 2.4% (PHOENIX14-T Dev) and 5.7% (CSL-Daily Dev).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CSGC instead of the strongest reported baseline | CSLR · PHOENIX14 Dev | WER: 19.7 -> 18.1 | Helped |
| Use CSGC instead of the strongest reported baseline | CSLR · PHOENIX14-T Dev | WER: 19.6 -> 17.2 | Helped |
| Use CSGC instead of the strongest reported baseline | CSLR · CSL-Daily Dev | WER: 32.8 -> 27.1 | Helped |
| Replace VAC with the cleaned baseline by removing its alignment constraint and last temporal pooling layer (ablation) | VAC on PHOENIX14 · PHOENIX14 | WER: 21.2 -> 20.7; 22.3 -> 21.1 | Helped |
| Add a global dataset-scope prototype rather than using no prototype (ablation) | baseline-clean on PHOENIX14 · PHOENIX14 | WER: 20.7 -> 19.2; 21.1 -> 19.7 | Helped |
| Restrict the prototype to sentence scope rather than using no prototype (ablation) | baseline-clean on PHOENIX14 · PHOENIX14 | WER: 20.7 -> 20.0 | Helped |
| Initialize gloss prototypes from dataset statistics rather than zeros or random values (ablation) | CSGC on PHOENIX14 · PHOENIX14 | WER: 19.7 -> 19.2; 19.6 -> 19.2 | Helped |
| Use momentum updates rather than global averaging for prototypes (ablation) | CSGC on PHOENIX14 Dev · PHOENIX14 Dev | WER: 21.1 -> 19.2 | Helped |
| Set the prototype momentum β to 0.9 (ablation) | CSGC on PHOENIX14 Dev · PHOENIX14 Dev | WER: 21.1 -> 19.2 | Helped |
| Add negative prototypes to positive-only gloss prototype learning (ablation) | CSGC on PHOENIX14 · PHOENIX14 | WER: 19.2 -> 18.5 | Helped |
| Add prototype learning to the FC classifier (ablation) | CSLR on PHOENIX14 · PHOENIX14 | WER: 20.7 -> 18.5; 21.1 -> 19.4 | Helped |
| Add the auxiliary similarity classifier to FC + GP (ablation) | CSLR on PHOENIX14 · PHOENIX14 | WER: 18.5 -> 18.1; 19.4 -> 19.0 | Helped |
Surprising findings
- Sentence-scope prototypes already improve over no prototype, and the global dataset-scope prototype gives further gains.
- Summing the momentum-averaged features without momentum weighting degrades performance relative to momentum update.
Limitations
- Gloss prototypes depend on CTC pseudo-labels from the intra-sentence classifier, so errors propagate into the prototype memory.
- Gains are reported only with a ResNet18 visual-only backbone and BiLSTM setup on three datasets.
Experimental setup
Datasets: PHOENIX14 PHOENIX14-T CSL-Daily
Benchmarks: PHOENIX14 PHOENIX14-T CSL-Daily
Metrics: WER Gloss Accuracy
Baselines: VAC SMKD C2SLR TLP SFL STMC SignBT Joint-SLRT
Builds on: VAC InfoNCE CTC
Closest papers by meaning
- Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation (NeurIPS 2025 · similarity 0.87)
- Improving Gloss-free Sign Language Translation by Reducing Representation Density (NeurIPS 2024 · similarity 0.86)
- TCNet: Continuous Sign Language Recognition from Trajectories and Correlated Regions (AAAI 2024 · similarity 0.85)
- Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment (AAAI 2024 · similarity 0.85)
- Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · similarity 0.84)
- SignGraph: A Sign Sequence is Worth Graphs of Nodes (CVPR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.