Research map / Large language models / Language, safety and interpretability
Scaling Sign Language Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales sign language translation via large-scale pretraining on noisy multilingual YouTube SLT data, MADLAD MT data, and MT-augmented SLT data, initializing (m/By)T5 encoder-decoder models, then finetuning on five open-domain SLT benchmarks to reach new SOTA.
Problem
Sign language translation remains limited to narrow domains and few sign languages, and existing methods struggle on open-domain benchmarks.
Approach
Scale SLT with noisy multilingual YouTube SLT data, multilingual MT data, and MT-augmented multiway SLT data within a unified prompted encoder-decoder initialized from T5/mT5/ByT5 across Base/Large/XL sizes, using clip-level multi-task pretraining on pose landmarks.
Key results
- Final pretrained-then-finetuned model reaches average BLEURT 49.60 (BLEU 16.90, ChrF 39.49) across downstream benchmarks, surpassing previous SOTA on all benchmarks.
- Augmented SLT plus multilingual MT with ByT5 XL raises FLEURS-ASL#0 average BLEURT from 29.52 to 36.01 (Base) and then to 45.12 (XL) for Small target languages.
- Spearman correlation between pretraining and finetuning results is 0.578 for BLEURT versus 0.347 for BLEU and 0.186 for ChrF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase SLT pretraining data from YT-ASL to YT-Full (ablation) | ByT5-based SLT · How2Sign and FLEURS-ASL#0 (En) | BLEURT: gain reaches ~7 BLEURT on How2Sign and ~11 BLEURT on FLEURS-ASL#0 (En) for Base and XL, respectively | Helped |
| Use ByT5 instead of T5 or mT5 (ablation) | SLT pretraining · How2Sign and FLEURS-ASL#0 (En) | BLEURT | Helped |
| Increase model size from Base to Large or XL without adding more data (ablation) | SLT pretraining · How2Sign and FLEURS-ASL#0 (En) | BLEURT | Mixed |
| Add bilingual MT data to SLT pretraining (ablation) | ByT5 Base with YT-ASL · How2Sign and FLEURS-ASL#0 (En) | BLEURT: +3 BLEURT | Helped |
| Use Ja→En MT instead of no MT data (ablation) | ByT5 Base with YT-ASL · How2Sign and FLEURS-ASL#0 (En) | BLEURT: at least 3 BLEURT | Helped |
| Use Fr→En MT instead of no MT data (ablation) | ByT5 Base with YT-ASL · FLEURS-ASL#0 (En) | BLEURT | Mixed |
| Use X→En rather than X↔En MT data (ablation) | ASL-to-En SLT · How2Sign and FLEURS-ASL#0 (En) | BLEURT: +0.08 and +0.9 BLEURT | Helped |
| Increase the MT sampling ratio from p_mt=0.5 toward p_mt=0.9 (ablation) | SLT pretrained with En-De MT · How2Sign and FLEURS-ASL#0 (En/De) | BLEURT | Helped |
| Add multilingual En→X or En↔X MT data (ablation) | Multilingual ASL SLT · FLEURS-ASL#0 (Small and Large target languages) | BLEURT | Helped |
| Add multilingual X→En MT data (ablation) | Multilingual ASL SLT with YT-Full · FLEURS-ASL#0 (Small and Large target languages) | BLEURT | Hurt |
| Augment SLT captions by translating them into additional spoken languages (ablation) | ByT5 Base multilingual SLT · FLEURS-ASL#0 (Small and Large target languages) | BLEURT: 29.52→36.01; 30.69→39.85 | Helped |
| Increase model capacity from ByT5 Base to ByT5 XL with augmented SLT and MT (ablation) | Multilingual SLT · FLEURS-ASL#0 (Small and Large target languages) | BLEURT: 36.01→45.12; 39.85→48.05 | Helped |
Surprising findings
- Base models often outperform Large and XL in pretraining, so model scaling alone does not consistently help SLT.
- Zero-shot ASL-to-X translation via joint ASL-to-En SLT and En-X MT works but at poor quality, with unstable gains across languages.
Limitations
- Zero-shot SLT quality is poor and unstable across languages.
- Models are limited to (m/By)T5 encoder-decoder architectures and SLT pretraining requires substantial compute.
- Multilingual benchmarks are lacking, and the models may show demographic biases.
Experimental setup
Datasets: YouTube-ASL YT-Full noisy multilingual YouTube SLT data MADLAD-400 How2Sign Elementary23 WMT23 DSGS WMT23 LIS-CH WMT23 LSF-CH FLEURS-ASL#0 FLORES-200
Benchmarks: How2Sign Elementary23 WMT23 DSGS SRF/SS WMT23 LIS-CH WMT23 LSF-CH FLEURS-ASL#0
Metrics: BLEURT BLEU ChrF
Baselines: SLTUNet previous SOTA on How2Sign previous SOTA on Elementary23 previous SOTA on WMT23 DSGS/LIS-CH/LSF-CH cascading ASL-to-En SLT plus MT
Builds on: SLTUNet T5 mT5 ByT5 MADLAD-400
Closest papers by meaning
- Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.94)
- LLMs are Good Sign Language Translators (CVPR 2024 · similarity 0.93)
- Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · similarity 0.93)
- YouTube-SL-25: A Large-Scale, Open-Domain Multilingual Sign Language Parallel Corpus (ICLR 2025 · similarity 0.93)
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation (ICLR 2024 · similarity 0.93)
- Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment (AAAI 2024 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.