Research map / Vision-language and multimodal / Cross-modal learning
Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CV-SLT, a conditional VAE framework with prior and posterior paths and two KL terms, plus an attention residual Gaussian distribution, for gloss-free sign language translation that aligns video and text directly.
Problem
Sign language translation suffers from a modality gap between sign videos and spoken text; prior methods rely on gloss supervision or leave visual-textual alignment insufficient.
Approach
A CVAE with a visual-only prior path and a visual+text posterior path, regularized by a KL term on encoder outputs and a self-distillation KL from posterior to prior on decoder outputs; the posterior is parameterized as a residual Gaussian relative to the prior via shared attention (ARGD). Gloss is not used.
Key results
- On PHOENIX14T test, CV-SLT reaches BLEU-4 29.27 vs 27.95 for MMTLB under the same gloss-free setup (+1.32).
- On CSL-daily test, CV-SLT reaches BLEU-4 28.94, improving over the MMTLB reproduction (27.30) by +1.64.
- Removing the AEP loss collapses performance (test BLEU-4 5.34), while removing ARGD drops test BLEU-4 from 29.27 to 26.67.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the alignment-enhanced prior path loss L_AEP (ablation) | CV-SLT · PHOENIX14T | BLEU-4: 4.74 -> 29.10 (Dev); 5.34 -> 29.27 (Test) | Helped |
| add self-distillation loss L_SD (ablation) | CV-SLT · PHOENIX14T | BLEU-4: 27.41 -> 29.10 (Dev); 28.59 -> 29.27 (Test) | Helped |
| use ARGD instead of a vanilla absolute posterior (ablation) | CV-SLT · PHOENIX14T | BLEU-4: 27.63 -> 29.10 (Dev); 26.67 -> 29.27 (Test) | Helped |
| use shared rather than non-shared attention (ablation) | CV-SLT · PHOENIX14T Dev and Test | BLEU-4: 28.59 -> 29.10 (Dev); 29.24 -> 29.27 (Test) | Mixed |
Surprising findings
- Removing the prior-path AEP loss causes near-total collapse (BLEU-4 around 5), attributed to KL vanishing and training-inference mismatch.
- The authors note a bug in MMTLB's full-width/half-width character handling on CSL-daily and re-evaluated it, so baseline numbers differ from the original paper.
Limitations
- Uses continuous latent variables only; discrete latents are left for future work.
- More advanced inference (Monte Carlo, importance sampling) was not used due to computational limits.
- Gains over MMTLB on PHOENIX14T dev are modest (about +1.4 to +1.9 BLEU).
Experimental setup
Datasets: PHOENIX14T CSL-daily
Benchmarks: PHOENIX14T CSL-daily
Metrics: BLEU-1 BLEU-2 BLEU-3 BLEU-4 ROUGE
Baselines: SL-Luong TSPNet Joint-SLRT MMTLB SignBT SMTC-T SLTUNET TS-SLT
Builds on: MMTLB conditional variational autoencoder transformer
Closest papers by meaning
- Scaling Sign Language Translation (NeurIPS 2024 · similarity 0.92)
- Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment (NeurIPS 2026 · similarity 0.92)
- LLMs are Good Sign Language Translators (CVPR 2024 · similarity 0.91)
- Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation (NeurIPS 2025 · similarity 0.91)
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation (ICLR 2024 · similarity 0.90)
- Improving Gloss-free Sign Language Translation by Reducing Representation Density (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.