Research map / Video / Video-language understanding
ReSCUE: Re-translation with Sentence Commitment for Unsegmented Long-Form Simultaneous Sign Language Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReSCUE, a simultaneous sign language translation framework for unsegmented long-form video that combines inference-aware training, stabilized re-translation (biased search and Mask-k), and sentence commitment via terminal punctuation.
Problem
Simultaneous sign language translation is confined to pre-segmented sentence-level clips, leaving continuous unsegmented long-form sign video streams without low-latency, stable, and boundary-aware translation.
Approach
Adapts speech-style re-translation to SLT with biased beam search and Mask-k for output stability, trains with prefix, silence-augmented (<SIL> token), and multi-sentence instances, and uses terminal punctuation plus token arrival times to commit sentences and advance the input buffer.
Key results
- On CSL-Daily, ReSCUE (k=0) reaches 693 ms AL versus 1163 ms for the strongest simultaneous baseline (CTL++ m=0), a 40.4% latency reduction.
- On CSL-Daily, ReSCUE (k=6) reaches 24.72 BLEU, improving over existing simultaneous SLT methods by 19.8% BLEU and 29.6% ROUGE.
- On CSL-Story, ReSCUE achieves 23.64 BLEU at 1588 ms AL versus 2.10 BLEU and 37722 ms AL for offline Uni-Sign applied to long-form video.
Surprising findings
- Removing silence or prefix-to-prefix training yields negative AL values because hallucinated or prematurely committed outputs are longer than the reference, making AL misleading.
- Offline Uni-Sign degrades sharply on long-form video (BLEU 0.51 on How2Sign-Long) without sentence boundaries.
Limitations
- Premature commitment on terminal punctuation from short prefixes, especially without prefix-to-prefix training.
- Delayed commitment when punctuation is not predicted, causing buffer growth and multiple sentences merged.
- Silence detection errors can discard signing frames near sentence starts; inherits pose estimation and out-of-vocabulary sign errors.
Experimental setup
Datasets: CSL-Daily Phoenix-2014-T How2Sign CSL-Story CSL-News YouTube-ASL
Benchmarks: CSL-Daily Phoenix-2014-T How2Sign-Long CSL-Story
Metrics: BLEU ROUGE-L AL LAAL Normalized Erasure
Baselines: SimulSLT CTL++ Uni-Sign CV-SLT SLRT GFSLT-VLP SignCL
Builds on: Uni-Sign re-translation Wait-k CTL++
Closest papers by meaning
- Scaling Sign Language Translation (NeurIPS 2024 · similarity 0.91)
- Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.89)
- Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · similarity 0.89)
- Uni-Sign: Toward Unified Sign Language Understanding at Scale (ICLR 2025 · similarity 0.88)
- LLMs are Good Sign Language Translators (CVPR 2024 · similarity 0.88)
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.