Research map / Video / Video-language understanding
Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PGG-SLT, a gloss-free SLT framework that prompts an LLM with in-context text-gloss examples to produce pseudo glosses, reorders them via weakly supervised multi-label classification, and trains a three-stage CTC-aided SLT model.
Problem
Gloss-based sign language translation needs expensive expert gloss annotations that are rarely available, while existing gloss-free methods lag behind; pseudo glosses from heuristics are misaligned with sign order, which hinders CTC supervision.
Approach
Use an LLM with a few text-gloss in-context examples to generate draft pseudo glosses from spoken text, then reorder them to match video temporal structure using a weakly supervised multi-label frame classifier with smoothing and frequency-aware weighting, enabling CTC-based Sign2Gloss and Gloss2Text pre-training.
Key results
- On Phoenix14T test, PGG-SLT (Gemma2) reaches BLEU4 27.32 vs. 23.09 for FLa-LLM and 26.02 for VAP among gloss-free methods, and is competitive with gloss-based MMTLB (28.39).
- On How2Sign test, PGG-SLT (Gemma2) achieves BLEU4 13.7, improving over SSVP-SLT's 7.0 (6.1 BLEU4 gain as reported) without large-scale pre-training.
- Reordering raises gloss BLEU4 on Phoenix14T train from 14.3 to 17.4 and PHX test translation BLEU4 from 26.0 to 26.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 30 in-context text-gloss pairs instead of no examples for pseudo-gloss generation (ablation) | Gemini1.5Pro pseudo gloss generation · Phoenix14T gloss test | BLEU4: 10.5 -> 13.8 | Helped |
| Use pseudo glosses generated with 30 in-context pairs instead of 0 pairs (ablation) | PGG-SLT · Phoenix14T test | BLEU4: 25.6 -> 26.2 | Helped |
| Increase in-context text-gloss examples from 30 to 700 (ablation) | Gemini1.5Pro pseudo gloss generation · Phoenix14T gloss test | BLEU4: 13.8 -> 17.2 | Helped |
| Replace generated pseudo glosses with true glosses (ablation) | SLT training · Phoenix14T test | BLEU4: 26.2 -> 28.2 | Helped |
| Use Gemini1.5Pro rather than Gemma2-7B for pseudo-gloss generation (ablation) | Pseudo-gloss generation · Phoenix14T gloss test | BLEU4: 11.9 -> 13.8 | Helped |
| Add weakly supervised pseudo-gloss reordering (ablation) | PGG-SLT · Phoenix14T | BLEU4: 26.0 -> 26.9 | Helped |
| Add weakly supervised pseudo-gloss reordering (ablation) | PGG-SLT · How2Sign | BLEU4: 12.6 -> 13.1 | Helped |
| Add temporal smoothing to weakly supervised classification (ablation) | Weakly supervised gloss classifier · Phoenix14T gloss test | BLEU4: 36.3 -> 41.4 | Helped |
| Add frequency-aware weighting alongside temporal smoothing (ablation) | Weakly supervised gloss classifier · Phoenix14T gloss test | BLEU4: 41.4 -> 43.1 | Helped |
| Use frequency-aware weighting without temporal smoothing (ablation) | Weakly supervised gloss classifier · Phoenix14T gloss test | BLEU4: 36.3 -> 36.1 | No clear effect |
| Use sentence-level supervision only, without gloss annotations or gloss-based stages (ablation) | Sign2Text · Phoenix14T | BLEU4: BLEU4 scores of just 8.2 | Hurt |
| Use sentence-level supervision only, without gloss annotations or gloss-based stages (ablation) | Sign2Text · How2Sign | BLEU4: BLEU4 scores of just 2.3 | Hurt |
Surprising findings
- Direct Sign2Text training with only sentence-level supervision yields poor BLEU4 (8.2 on Phoenix14T, 2.3 on How2Sign), and more epochs do not help.
- Even with zero in-context examples, LLM-generated glosses outperform the heuristic POS-based pseudo glosses by a large margin (BLEU4 10.5 vs 2.0 on test).
Limitations
- Relies on proprietary Gemini 1.5 Pro for pseudo gloss generation on the main setting, limiting reproducibility.
- Pseudo glosses remain noisy relative to true glosses (BLEU4 around 14 vs. ground truth), and the CTC loss is removed in late training to avoid over-reliance on noisy supervision.
- Some How2Sign pseudo glosses exceed video length and required manual removal of redundant words via a hand-crafted stopword list.
Experimental setup
Datasets: Phoenix14T How2Sign
Benchmarks: Phoenix14T How2Sign
Metrics: BLEU1 BLEU2 BLEU3 BLEU4 ROUGE WER
Baselines: MMTLB SLTUNet IP-SLT TS-SLT NSLT GASLT GFSLT-VLP SignLLM Sign2GPT FLa-LLM VAP SSVP-SLT
Builds on: MMTLB Sign2GPT GFSLT-VLP CTC mBART
Closest papers by meaning
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation (ICLR 2024 · similarity 0.93)
- Improving Gloss-free Sign Language Translation by Reducing Representation Density (NeurIPS 2024 · similarity 0.93)
- Scaling Sign Language Translation (NeurIPS 2024 · similarity 0.92)
- Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · similarity 0.92)
- Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.91)
- Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment (AAAI 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.