Research map / Large language models / Language, safety and interpretability
Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Translates continuous sign language to English by fine-tuning Llama3-8B with LoRA on visual sign features plus text cues (pseudo-glosses, previous-sentence translation, BLIP2 background keywords), evaluated on BOBSL and How2Sign.
Problem
Sign language translation to spoken text in the open-vocabulary setting is hard due to weak, noisy subtitle supervision and the context dependence of sign languages (referents, topic-comment structure, homonyms).
Approach
Feed a pre-trained LLM with visual sign features (via MLP mapping) together with textual contextual cues: pseudo-glosses from an ISLR classifier, the predicted previous sentence, and background keywords from an image captioner over the scene behind the signer, with augmentations (word dropping, cue dropping, GT/predicted previous sentence sampling).
Key results
- On BOBSL SENT-TEST, the full model (Vid+PrevGT+Spot+BG) reaches 47.1 BLEURT vs 37.0 for the Sincan et al. oracle, and the video-only model already beats Sign2GPT (37.8 vs 34.3 BLEURT).
- On the validation set, adding pseudo-glosses, previous predicted sentence, and background descriptions raises BLEURT from 41.0 to 43.5 and IoU from 16.6 to 18.8.
- On How2Sign test, the Vid+PG+PrevPred model reaches R-L 32.5 vs 27.8 for VAP, with comparable BLEU-4 (12.7 vs 12.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add pseudo-glosses to visual features (ablation) | Ours on BOBSL SENT-VAL · BOBSL SENT-VAL | BLEURT: 41.0 -> 41.8 | Helped |
| Add the predicted previous sentence (ablation) | Ours on BOBSL SENT-VAL · BOBSL SENT-VAL | BLEURT: 41.8 -> 42.5 | Helped |
| Add background descriptions (ablation) | Ours on BOBSL SENT-VAL · BOBSL SENT-VAL | BLEURT: 42.5 -> 43.5 | Helped |
| Apply word dropping to textual cues (ablation) | Ours on BOBSL SENT-VAL · BOBSL SENT-VAL | BLEURT: 41.2 -> 41.4 | Helped |
| Apply whole-cue dropping (ablation) | Ours on BOBSL SENT-VAL · BOBSL SENT-VAL | BLEURT: 41.4 -> 42.7 | Helped |
| Sample predicted or ground-truth previous sentences during training (ablation) | Ours on BOBSL SENT-VAL · BOBSL SENT-VAL | BLEURT: 42.7 -> 43.5 | Helped |
| Fine-tune the LLM with LoRA rather than keeping it frozen (ablation) | Ours on BOBSL SENT-VAL · BOBSL SENT-VAL | BLEURT: 40.6 -> 43.5 | Helped |
| Add pseudo-glosses and the predicted previous sentence (ablation) | Ours on How2Sign · How2Sign test | BLEURT: 44.1 -> 45.3 | Helped |
Surprising findings
- Background descriptions can sometimes hinder translation by introducing conflicting information.
- Previous-sentence predictions can cause error accumulation across sentences.
Limitations
- Errors accumulate from predicted previous sentences.
- Pseudo-glosses produce false positives from homonyms.
- Struggles with questions vs statements, negation, pointing and fingerspelling.
Experimental setup
Datasets: BOBSL How2Sign PHOENIX14T CSL-Daily
Benchmarks: BOBSL SENT-VAL BOBSL SENT-TEST How2Sign test
Metrics: BLEU-4 BLEURT ROUGE-L CIDEr IoU LLM score
Baselines: Albanie et al. (BOBSL SLT) Sincan et al. GFSLT Sign2GPT SSLT SSVP-SLT VAP Fla-LLM
Builds on: Llama3 Video-Swin LoRA BLIP-2 Sign2GPT
Closest papers by meaning
- Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.94)
- Scaling Sign Language Translation (NeurIPS 2024 · similarity 0.93)
- LLMs are Good Sign Language Translators (CVPR 2024 · similarity 0.92)
- Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation (NeurIPS 2025 · similarity 0.92)
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation (ICLR 2024 · similarity 0.91)
- SCOPE: Sign Language Contextual Processing with Embedding from LLMs (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.