Research map / Large language models / Language, safety and interpretability
LLMs are Good Sign Language Translators
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SignLLM, which quantizes sign videos into discrete character-level tokens, merges them into word-level tokens via an optimal-transport codebook construction, and feeds them to a frozen LLaMA-7B for gloss-free sign language translation.
Problem
Sign language translation (SLT) from sign videos to spoken language is hard due to limited paired sign-text data and the large gap between sign video features and text tokens, so making sign videos readable by off-the-shelf LLMs is the challenge.
Approach
Regularize sign videos into language-like representations: a VQ-based character-level codebook trained with context prediction, a word-level codebook built by entropy/size-aware optimal transport (Sinkhorn), and MMD-based sign-text alignment, all feeding a frozen LLM.
Key results
- On Phoenix-2014T test, SignLLM reaches BLEU-4 23.40 and ROUGE 44.49, outperforming gloss-free GFSLT-VLP (21.44 / 42.49).
- On CSL-Daily test, SignLLM reaches BLEU-4 15.75 and ROUGE 39.91 versus GFSLT-VLP's 11.00 / 36.44.
- Ablation on Phoenix-2014T test: removing sign-text alignment drops BLEU-4 from 23.40 to 10.63.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SignLLM instead of GFSLT-VLP | gloss-free SLT · Phoenix-2014T test | BLEU-4: 21.44 -> 23.40 | Helped |
| Use SignLLM instead of GFSLT-VLP | gloss-free SLT · CSL-Daily test | BLEU-4: 11.00 -> 15.75 | Helped |
| Replace LLaMA with mBART | SignLLM on Phoenix-2014T · Phoenix-2014T test | BLEU-4: 23.40 -> 10.36 | Hurt |
| Replace LLaMA with T5 | SignLLM on Phoenix-2014T · Phoenix-2014T test | BLEU-4: 23.40 -> 22.51 | Hurt |
| Feed visual encoder outputs directly to the frozen LLM (ablation) | SignLLM on Phoenix-2014T · Phoenix-2014T test | BLEU-4: 23.40 -> 8.42 | Hurt |
| Feed visual encoder outputs directly to the LLM and fine-tune with LoRA (ablation) | SignLLM on Phoenix-2014T · Phoenix-2014T test | BLEU-4: 23.40 -> 18.17 | Hurt |
| Replace VQ-Sign with k-means quantization (ablation) | SignLLM on Phoenix-2014T · Phoenix-2014T test | BLEU-4: 23.40 -> 13.13 | Hurt |
| Remove codebook reconstruction (ablation) | SignLLM on Phoenix-2014T · Phoenix-2014T test | BLEU-4: 23.40 -> 19.47 | Hurt |
| Remove sign-text alignment loss (ablation) | SignLLM on Phoenix-2014T · Phoenix-2014T test | BLEU-4: 23.40 -> 10.63 | Hurt |
Surprising findings
- Using a frozen LLM with SignLLM's sign tokens substantially outperforms LoRA fine-tuning of the LLM on encoder outputs (test B4 18.17 vs 23.40).
Limitations
- Gap to gloss-based SOTA remains on some metrics (e.g., Phoenix test BLEU-4 23.40 vs gloss-based methods around 28-29).
- Relies on a pre-trained ImageNet ResNet18 visual encoder and a relatively small SLT dataset.
Experimental setup
Datasets: Phoenix-2014T CSL-Daily
Benchmarks: Phoenix-2014T CSL-Daily
Metrics: BLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE-L
Baselines: GFSLT-VLP GASLT CSGCR TSPNet NSLT TS-SLT SLTUNET MMTLB
Builds on: GFSLT-VLP VQ-VAE contrastive predictive coding LLaVA
Closest papers by meaning
- Scaling Sign Language Translation (NeurIPS 2024 · similarity 0.93)
- Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.93)
- Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · similarity 0.92)
- Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment (AAAI 2024 · similarity 0.91)
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation (ICLR 2024 · similarity 0.91)
- Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.