Research map / Vision-language and multimodal / CLIP and zero-shot
Scaling up Multimodal Pre-Training for Sign Language Understanding
Wengang Zhou, Weichao Zhao, Hezhen Hu, Zecheng Li, Houqiang Li
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase SL-1.5M pre-training data from 0% to 100% (ablation) | Ours · MSASL1000 | Top-1 accuracy: 62.23 -> 74.07 | Improved |
| increase SL-1.5M pre-training data from 0% to 100% (ablation) | Ours · Phoenix14 | WER: 28.9 -> 21.2 | Improved |
| increase SL-1.5M pre-training data from 0% to 100% (ablation) | Ours · Phoenix14-T | BLEU-4: 13.15 -> 22.24 | Improved |
| add sign-text contrastive learning to masked pose modeling (ablation) | Ours · MSASL1000 | Top-1 accuracy: 69.44 -> 74.07 | Improved |
| add masked pose modeling to sign-text contrastive learning (ablation) | Ours · MSASL1000 | Top-1 accuracy: 67.16 -> 74.07 | Improved |
| add non-manual (face and mouth) features to manual features (ablation) | Ours · MSASL1000 | Top-1 accuracy: 70.24 -> 74.07 | Improved |
| use fine-grained sign-text similarity instead of coarse-grained (ablation) | Ours · CSL-Daily SL-RT | T2V R@1: 79.8 -> 87.5 | Improved |
| use GCN pose embedding instead of linear projector (ablation) | Ours · CSL-Daily SL-RT | T2V R@1: 77.4 -> 87.5 | Improved |
| set mask ratio to 40% instead of 0% (ablation) | Ours · MSASL1000 | Top-1 accuracy: 67.16 -> 74.07 | Improved |
| raise mask ratio from 40% to 80% (ablation) | Ours · MSASL1000 | Top-1 accuracy: 74.07 -> 66.36 | Hurt |
| enlarge sliding window from 4 to 16 (ablation) | Ours · Phoenix14-T | BLEU-4: 22.24 -> 18.17 | Hurt |
| set contrastive loss weight to 1.0 instead of 0.1 (ablation) | Ours · MSASL1000 | Top-1 accuracy: 71.74 -> 74.07 | Improved |
Closest papers by meaning
- Scaling Sign Language Translation (NeurIPS 2024 · similarity 0.92)
- CVT-SLR: Contrastive Visual-Textual Transformation for Sign Language Recognition With Variational Alignment (CVPR 2023 · similarity 0.92)
- Addressing Resource Scarcity across Sign Languages with Multilingual Pretraining and Unified-Vocabulary Datasets (NeurIPS 2022 · similarity 0.92)
- Uni-Sign: Toward Unified Sign Language Understanding at Scale (ICLR 2025 · similarity 0.91)
- Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.91)
- Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment (AAAI 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
