Research map / Large language models / Language, safety and interpretability
Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Sign2GPT adapts frozen DINOv2 ViT-S/14 and XGLM-1.7B via LoRA and zero-gated cross-attention for gloss-free sign language translation, with a pseudo-gloss prototype pretraining stage for the sign encoder.
Problem
Gloss-free sign language translation suffers from limited paired data, signer variation, and long video sequences that are memory-intensive to train end-to-end.
Approach
Keep large pretrained vision (DINOv2) and language (XGLM) models frozen and adapt them with lightweight LoRA and zero-gated cross-attention, while pretraining a sign encoder to detect spoken-language pseudo-glosses (extracted via spaCy POS filtering) against fastText prototypes using a binary cross-entropy presence objective, requiring no gloss annotations or sign order.
Key results
- On Phoenix14T, Sign2GPT(w/PGP) reaches 22.52 BLEU-4 and 48.90 ROUGE on test, versus 21.44 BLEU-4 for the prior gloss-free SOTA GFSLT-VLP.
- On CSL-Daily, Sign2GPT(w/PGP) reaches 15.40 BLEU-4 versus 11.00 for GFSLT-VLP, about 4.4 BLEU-4 improvement.
- Pseudo-gloss pretraining improves BLEU-4 on CSL-Daily from 12.96 to 15.40 and on Phoenix14T from 19.42 to 22.52.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add pseudo-gloss pretraining | Sign2GPT · Phoenix14T test | BLEU4: 19.42 -> 22.52 | Helped |
| Add pseudo-gloss pretraining | Sign2GPT · CSL-Daily test | BLEU4: 12.96 -> 15.40 | Helped |
| Add spatial adapters (ablation) | Sign2GPT · Phoenix14T dev | BLEU4: 16.38 -> 19.55 | Helped |
| Use local attention instead of global attention (ablation) | Sign2GPT · Phoenix14T dev | BLEU4: 18.56 -> 19.55 | Helped |
| Add temporal downsampling (ablation) | Sign2GPT · Phoenix14T dev | BLEU4: 19.30 -> 19.55 | Helped |
| Add sinusoidal positional embeddings (ablation) | Sign2GPT(w/PGP) · Phoenix14T dev | BLEU4: 21.68 -> 23.20 | Helped |
| Use learnable positional embeddings with zero initialization (ablation) | Sign2GPT(w/PGP) · Phoenix14T dev | BLEU4: 21.68 -> 21.89 | No clear effect |
| Use learnable positional embeddings with random initialization (ablation) | Sign2GPT(w/PGP) · Phoenix14T dev | BLEU4: 21.68 -> 21.16 | Hurt |
| Use DinoV2 (ViT-S/14) instead of ResNet18 (ablation) | Sign2GPT · CSL-Daily test | BLEU4: 12.28 -> 12.96 | Helped |
| Use selected pseudo-glosses instead of all words (ablation) | Pseudo-gloss pretraining · Phoenix14T | Recall: 0.28 -> 0.39 | Helped |
| Use selected pseudo-glosses instead of all words (ablation) | Pseudo-gloss pretraining · Phoenix14T | Precision: 0.55 -> 0.52 | Hurt |
| Add the sign encoder (ablation) | Pseudo-gloss pretraining · Phoenix14T | F1-Score: 0.33 -> 0.44 | Helped |
Surprising findings
- Training only the decoder's small trainable parameters on frozen pretrained sign features (Sign(Z)2GPT) remains competitive, with under 4 million trainable decoder parameters on Phoenix14T.
Limitations
- Pseudo-glosses are derived from spoken-language sentences via POS filtering, so their correspondence to actual signs is approximate (pretraining F1 only 0.44 on Phoenix14T and 0.36 on CSL-Daily).
- Only two benchmarks evaluated, both with sentence-level BLEU/ROUGE metrics.
- CSL-Daily requires vocabulary expansion with 252 new frozen tokens for unseen Chinese characters.
Experimental setup
Datasets: RWTH-PHOENIX-Weather 2014T CSL-Daily
Benchmarks: RWTH-PHOENIX-Weather 2014T CSL-Daily
Metrics: BLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE-L
Baselines: SL-Transformer BN-TIN-Transf.+BT MMTLB SLTU-NET TwoStream-SLT NSLT TSPNet CSGCR GASLT GFSLT GFSLT-VLP
Builds on: LoRA DINOv2 XGLM GFSLT-VLP fastText
Closest papers by meaning
- Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation (NeurIPS 2025 · similarity 0.93)
- Improving Gloss-free Sign Language Translation by Reducing Representation Density (NeurIPS 2024 · similarity 0.93)
- Scaling Sign Language Translation (NeurIPS 2024 · similarity 0.93)
- Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · similarity 0.91)
- LLMs are Good Sign Language Translators (CVPR 2024 · similarity 0.91)
- Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.