Research map / Large language models / Language, safety and interpretability

Lost in Translation, Found in Embeddings: Sign Language Translation and Alignment

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Sign and cross-lingual translation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds a unified sign language model that performs both sign language translation and sign-subtitle alignment using a lightweight pose+lip visual backbone, a Sliding Perceiver mapper, and a Flan-T5-XL LLM, pretrained on BSL/ASL data.

Problem

Sign language translation (SLT) and sign-subtitle alignment (SSA) are usually handled separately; existing visual backbones are large, frozen, and trained on small, few-signer datasets, and SSA annotation is costly.

Approach

A lightweight keypoint (MediaPipe pose/hands/face) plus lip-reader backbone trained end-to-end with an LLM, a Sliding Perceiver that compresses local windows of frames into word-level embeddings, and a multitask prompt-based SLT+SSA objective with soft-decoded timestamp L1 loss.

Key results

• On BOBSL-SENT test, Ours (Multitask) reaches B4 6.8 and CIDEr 77.0 versus LiTFiC (Full) 3.3 B4 and 41.9 CIDEr.
• On How2Sign test after finetuning, the model reaches B4 24.6 and BLEURT 58.7, surpassing prior methods including SSVP-SLT (B4 15.5 before this model's finetuning comparisons listed).
• On BOBSL-SSA test, Ours (Multitask) attains 77.1 accuracy, 67.5 CB-Acc and F1@0.50 of 68.9, versus SAT+ at 77.2, 65.0 and 63.8.

Surprising findings

• Pose-only backbone matches the Video-Swin ISLR baseline on top-1 per-instance accuracy while using less than half the GFLOPs.
• Zero-shot How2Sign BLEURT of 51.3 is comparable to Uni-Sign and Geo-Sign, which were pretrained on Chinese Sign Language and finetuned on How2Sign.

Limitations

• Fixed 20-second input window cannot cover subtitles or audio-signing lags longer than the window.
• Model mispredicts context-dependent content (e.g., background objects) that requires broader context.
• Large-scale pretraining restricted to English-related sign languages (BSL and ASL).

Experimental setup

Datasets: BOBSL YouTube-SL-25 How2Sign FLEURS-ASL

Benchmarks: BOBSL-SENT BOBSL-SIGN How2Sign FLEURS-ASL

Metrics: BLEU-4 BLEURT ROUGE-L CIDEr IoU LLM score frame accuracy class-balanced accuracy F1@0.10 F1@0.25

Baselines: LiTFiC GFSLT Sign2GPT Albanie et al. Sincan et al. SAT SAT+ Video-Swin-ISLR Token Shuffle SSLT SSVP-SLT Uni-Sign

Builds on: Perceiver LiTFiC SAT Flan-T5 DoRA Conformer MediaPipe

Closest papers by meaning

• Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · similarity 0.94)
• Scaling Sign Language Translation (NeurIPS 2024 · similarity 0.94)
• LLMs are Good Sign Language Translators (CVPR 2024 · similarity 0.93)
• Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment (AAAI 2024 · similarity 0.92)
• Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation (NeurIPS 2025 · similarity 0.91)
• Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation (ICLR 2024 · similarity 0.91)