Research map / Video / Video-language understanding
From Clips to Streams: A Unified Framework for Streaming Sign Language Translation
Hoang Quan Dang, Xin Shen, Yanbin Liu, Xin Yu, Ling Chen
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Defines Streaming Sign Language Translation, which requires discovering and translating linguistic events from continuous untrimmed sign video rather than pre-segmented clips.
Problem
Sign language translation research has focused on pre-segmented clips, which disconnects models from continuous real-world signing; the paper targets translating from untrimmed streaming video.
Approach
Formally defines Streaming SLT as the joint discovery and translation of linguistic events from continuous, untrimmed video, and proposes a unified framework to address it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train translation jointly with segmentation (StreamSLST) instead of a cascaded segment-then-translate pipeline | StreamSLST · BOBSL | BLEU-4: 0.02 -> 0.23 | Improved |
| joint training instead of cascaded pipeline | StreamSLST · Streaming-Phoenix-2014T | BLEU-4: 5.86 -> 8.16 | Improved |
| joint training instead of cascaded pipeline | StreamSLST · How2Sign | BLEU-4: 1.80 -> 2.00 | Improved |
| joint training instead of cascade | StreamSLST · Streaming-CSL-Daily | METEOR: 9.46 -> 7.34 | Hurt |
| use mBART decoder instead of deformable LSTM (ablation) | StreamSLST · Streaming-CSL-Daily | localization F1: 46.84 -> 59.37 | Improved |
| add localized visual features to query for translation (ablation) | StreamSLST · Streaming-Phoenix-2014T | BLEU-4: 2.89 -> 8.16 | Improved |
| let translation loss update shared queries, encoder and backbone (ablation) | StreamSLST · Streaming-Phoenix-2014T | BLEU-4: 5.81 -> 8.16 | Improved |
| add Tri-modal Stage-1 pretraining (ablation) | StreamSLST · Streaming-Phoenix-2014T | BLEU-4: 3.65 -> 8.16 | Improved |
| add Tri-modal Stage-1 pretraining (ablation) | StreamSLST · Streaming-Phoenix-2014T | localization F1: 29.90 -> 55.95 | Improved |
| add Tri-modal Stage-1 pretraining (ablation) | StreamSLST · How2Sign | BLEU-4: 0.95 -> 2.00 | Improved |
| use learned proposals instead of annotated oracle spans (ablation) | StreamSLST · Streaming-Phoenix-2014T | BLEU-4: 3.08 -> 8.16 | Improved |
| widen oracle spans by 0.5 s / 1.0 s or rescale by 1.40 / 2.25 (ablation) | oracle-boundary StreamSLST · Streaming-Phoenix-2014T and How2Sign | BLEU-4: Phoenix 3.08 -> 3.79 at best vs 8.16 learned | No clear effect |
Closest papers by meaning
- ReSCUE: Re-translation with Sentence Commitment for Unsegmented Long-Form Simultaneous Sign Language Translation (NeurIPS 2026 · similarity 0.91)
- Scaling Sign Language Translation (NeurIPS 2024 · similarity 0.91)
- Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.90)
- Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · similarity 0.90)
- Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation (NeurIPS 2025 · similarity 0.89)
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
