Research map / Large language models / Language, safety and interpretability
SCOPE: Sign Language Contextual Processing with Embedding from LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SCOPE, a context-aware vision-based sign language recognition and translation framework that aligns pose-motion embeddings with frozen LLM context embeddings and fine-tunes Qwen2 with Q-LoRA, plus a 72.4-hour Chinese dialogue sign language dataset.
Problem
Vision-based sign language recognition and translation perform poorly on dialogue scenes because existing datasets are per-sentence and methods ignore preceding conversational context.
Approach
Align motion features from DWPose keypoints to frozen-LLM sentence embeddings of the target text, then condition gloss recognition on embeddings of the previous three dialogue sentences, and use the top-3 gloss hypotheses plus dialogue context as prompts to a Q-LoRA fine-tuned Qwen2 translator.
Key results
- On the SCOPE dataset, the full model reaches 27.4% WER on test, versus 30.7% without context and 33.8% for CorrNet.
- On the SCOPE test set, SLT BLEU-4 reaches 38.59 for the full model, versus 35.80 without context and 37.82 for CV-SLT.
- A 40-person Deaf community survey rated the application 4.15/5 for UX, 3.96/5 for SLR accuracy and 3.98/5 for SLT results.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add dialogue context to SCOPE-SLR (ablation) | SCOPE-SLR · SCOPE test | WER: 30.7 -> 27.4 | Helped |
| Add dialogue context to SCOPE-SLT (ablation) | SCOPE-SLT · SCOPE test | BLEU-4: 35.80 -> 38.59 | Helped |
| Add the embedding alignment encoder (ablation) | SCOPE · SCOPE test | BLEU-4: 22.58 -> 38.59 | Helped |
| Add the embedding alignment encoder (ablation) | SCOPE · SCOPE test | WER: 31.8 -> 27.4 | Helped |
| Add MWER loss (ablation) | SCOPE · SCOPE test | WER: 37.6 -> 27.4 | Helped |
| Add MWER loss (ablation) | SCOPE · SCOPE test | BLEU-4: 15.55 -> 38.59 | Helped |
| Add Iris Normalization (ablation) | SCOPE · SCOPE test | WER: 35.8 -> 27.4 | Helped |
| Add Iris Normalization (ablation) | SCOPE · SCOPE test | BLEU-4: 21.76 -> 38.59 | Helped |
| Use SCOPE-SLR instead of CorrNet | SLR · SCOPE test | WER: 33.8 -> 27.4 | Helped |
| Use SCOPE-SLT instead of CV-SLT | SLT · SCOPE test | BLEU-4: 37.82 -> 38.59 | Helped |
Surprising findings
- Adding the Qwen2 contextual fine-tuning gave the SLT gains, but removing the embedding alignment encoder dropped SLR performance and roughly halved BLEU-4 (to 22.58), which the authors attribute to difficulty aligning motion with LLM embeddings.
Limitations
- Dataset covers limited professional scenarios such as emergencies, insurance, and transportation.
- Very long dialogue contexts may have minimal impact on the current sentence; no attention decay mechanism is provided.
- Sign speed variation across signers is not handled by the recognition network.
Experimental setup
Datasets: PHOENIX-2014 PHOENIX-2014T CSL-Daily SCOPE
Benchmarks: PHOENIX-2014 PHOENIX-2014T CSL-Daily SCOPE
Metrics: WER BLEU ROUGE-L
Baselines: SEN-CSLR TwoStream-SLR CorrNet MMTLB-S2T TwoStream-S2T CV-SLT SlowFastSign
Builds on: CTC MWER training Q-LoRA DWPose CV-SLT
Closest papers by meaning
- Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · similarity 0.89)
- Scaling Sign Language Translation (NeurIPS 2024 · similarity 0.89)
- LLMs are Good Sign Language Translators (CVPR 2024 · similarity 0.88)
- Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation (NeurIPS 2025 · similarity 0.88)
- Uni-Sign: Toward Unified Sign Language Understanding at Scale (ICLR 2025 · similarity 0.88)
- KD-MSLRT: Lightweight Sign Language Recognition Model Based on Mediapipe and 3D to 1D Knowledge Distillation (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.