Research map / Large language models / Language, safety and interpretability
Gloss-Free Sign Language Translation: Improving from Visual-Language Pretraining
Benjia Zhou, Zhigang Chen, Albert Clapés, Jun Wan, Yanyan Liang, Sergio Escalera, Zhen Lei, Du Zhang
This paper tried a common change: Add data augmentation
add strong data augmentation in stage 1 on top of VLP · PHOENIX14T test · BLEU-4: 19.77 -> 21.23 Improved
What 689 accepted papers report for this change
- 560 improved
- 44 no clear effect
- 26 mixed
- 59 hurt
- Understanding Multimodal Learning: A Loss Landscape Smoothness Perspective ICML 2026 · add data augmentation (jitter, crop, flip) · UPMC-Food101 · accuracy: 92.84 -> 94.93Improved
- AsyncBEV: Cross-modal flow alignment in Asynchronous 3D Object Detection ICLR 2026 · add EMC + asynchronous data augmentation training · nuScenes val, 0 s offset · NDS: 66.7 -> 63.1Hurt
- Towards Understanding Continual Factual Knowledge Acquisition of Language Models: From Theory to Algorithm ICML 2026 · add data augmentation (5 biographies per individual) in pre-training · Biography PT test, unseen templates · EM: 2.71 -> 81.27Improved
See this change in detailMatched by the wording of each paper's own result, read by AI from the full text. Papers report gains more often than failures, so this shows what gets published, not the chance a change works.
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add visual-language pretraining (CLIP-style plus masked text) with strong augmentation (ablation) | GFSLT · PHOENIX14T test | BLEU-4: 19.66 -> 21.44 | Improved |
| add VLP with only lightweight augmentation (random crop) (ablation) | GFSLT · PHOENIX14T test | BLEU-4: 19.66 -> 19.77 | No clear effect |
| add strong data augmentation in stage 1 on top of VLP (ablation) | GFSLT-VLP · PHOENIX14T test | BLEU-4: 19.77 -> 21.23 | Improved |
| add strong data augmentation in stage 2 without VLP (ablation) | GFSLT · PHOENIX14T test | BLEU-4: 19.66 -> 18.41 | Hurt |
| fine-tune Visual Embedding and Transformer Encoder jointly from pretrained weights (ablation) | GFSLT · PHOENIX14T test | BLEU-4: 19.66 -> 21.43 | Improved |
| initialize only Visual Embedding from pretraining (ablation) | GFSLT · PHOENIX14T test | BLEU-4: 19.66 -> 18.93 | Hurt |
| initialize Text Decoder from pretraining (ablation) | GFSLT · PHOENIX14T test | BLEU-4: 19.66 -> 20.27 | Improved |
| train stage 2 for 200 instead of 40 epochs (ablation) | GFSLT-VLP · PHOENIX14T | BLEU-4: 18.23 -> 22.12 | Improved |
| increase stage 1 pretraining beyond 80 epochs (ablation) | GFSLT-VLP · PHOENIX14T | BLEU-4: 22.12 -> 22.13 | No clear effect |
Closest papers by meaning
- Improving Gloss-free Sign Language Translation by Reducing Representation Density (NeurIPS 2024 · similarity 0.95)
- Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation (NeurIPS 2025 · similarity 0.95)
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation (ICLR 2024 · similarity 0.94)
- Scaling Sign Language Translation (NeurIPS 2024 · similarity 0.93)
- Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment (AAAI 2024 · similarity 0.92)
- Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
