Research map / Large language models / Instruction and fine-tuning
Uni-Sign: Toward Unified Sign Language Understanding at Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uni-Sign pre-trains a pose-plus-RGB sign language model generatively on a new 1,985-hour CSL-News corpus with an mT5-Base decoder, then fine-tunes every SLU task (ISLR, CSLR, SLT) as a single language-modeling translation task.
Problem
Sign language understanding pre-training suffers from a gap with downstream fine-tuning, since pretext tasks are not aligned with text generation, and CSL has little large-scale data.
Approach
Generative pre-training of a sign encoder into a pretrained LLM (mT5-Base) with one language-modeling loss, with every downstream SLU task reformulated as translation-style generation, plus a prior-guided fusion module using keypoints as deformable-attention references and score-aware sampling of RGB frames.
Key results
- On OpenASL, Uni-Sign (RGB-pose) reaches 23.14 BLEU4 on test, beating C$^2$RL's 13.21 and a BLEURT of 60.40 vs 36.35.
- On CSL-Daily SLT, BLEU4 improves by 14.02 (dev) and 4.75 (test) over the previous gloss-free SOTA.
- On MSASL1000 ISLR, per-instance Top-1 improves 4.09 points over the previous best (MSLU); the pose-only model is 93.26 on MSASL100 and 92.24 on WLASL100.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use unified generative fine-tuning instead of task-specific ISLR classification and CSLR CTC fine-tuning (ablation) | Uni-Sign · CSL-Daily CSLR | WER: 28.2 vs. 37.4 and 27.4 vs. 36.4 | Helped |
| use language-model encoder features rather than temporal encoder features with task-specific fine-tuning (ablation) | Uni-Sign · MSASL1000 ISLR | P-I Top-1 accuracy: 6.91% | Hurt |
| increase CSL-News pre-training data quantity (ablation) | Uni-Sign · MSASL1000 and CSL-Daily | task performance | Helped |
| increase RGB sampling probability to 50% (ablation) | Uni-Sign score-aware sampling · CSL-Daily test | BLEU4: +1.25 BLEU4 | Helped |
| replace deformable attention with cross-attention in PGF (ablation) | Uni-Sign · CSL-Daily CSLR and SLT | performance | Hurt |
| add body keypoints to hands (ablation) | Uni-Sign pose-only model · MSASL1000 ISLR | P-I Top-1 accuracy: 36.07 -> 41.56 | Helped |
| add face keypoints to hands and body (ablation) | Uni-Sign pose-only model · MSASL1000 ISLR | P-I Top-1 accuracy: 41.56 -> 43.45 | Helped |
| add face keypoints to hands and body (ablation) | Uni-Sign pose-only model · CSL-Daily CSLR dev | WER: 53.0 -> 51.2 | Helped |
| add RGB features using the RGB-pose setting (ablation) | Uni-Sign · CSL-Daily SLT test | BLEU4: 25.61 -> 26.36 | Helped |
| add RGB features using the RGB-pose setting (ablation) | Uni-Sign · OpenASL test | BLEU4: 22.67 -> 23.14 | Helped |
Surprising findings
- Uni-Sign without the CSLR-specific CTC loss still trails TS-SLR by only 1.3% and 0.7% on CSLR.
- Using the unified LM fine-tuning on CSLR gives WER 28.2 vs 37.4 for a CTC-based setup, showing the generative objective transfers well.
Limitations
- No finely annotated, open-domain, large-scale benchmark is available to probe capabilities and limitations.
- All parameters are fine-tuned in every stage, which is computationally demanding.
- Fails to disambiguate semantically similar words and complex sentence structures.
Experimental setup
Datasets: CSL-News YouTube-ASL WLASL MSASL CSL-Daily How2Sign OpenASL
Benchmarks: WLASL MSASL CSL-Daily How2Sign OpenASL
Metrics: per-instance Top-1 accuracy per-class Top-1 accuracy word error rate BLEU ROUGE-L BLEURT
Baselines: SignBERT+ MSLU NLA-SLR TS-SLR C$^2$RL SignLLM Sign2GPT FLa-LLM SSVP-SLT YouTube-ASL GloFE-VN TS-SLT
Builds on: mT5 ST-GCN deformable attention GFSLT-VLP YouTube-ASL
Closest papers by meaning
- Scaling Sign Language Translation (NeurIPS 2024 · similarity 0.91)
- Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.90)
- KD-MSLRT: Lightweight Sign Language Recognition Model Based on Mediapipe and 3D to 1D Knowledge Distillation (AAAI 2025 · similarity 0.90)
- Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · similarity 0.89)
- Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment (AAAI 2024 · similarity 0.88)
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.