Research map / Video / Video segmentation and tracking
Discrete to Continuous: Generating Smooth Transition Poses from Sign Language Observations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sign-D2C, a conditional diffusion model that generates smooth transition poses between discrete sign segments, trained by randomly masking long sign pose sequences and inferred with linear-interpolation padding.
Problem
Concatenating isolated signs into continuous sign language sequences yields abrupt, unnatural transitions, and there is no supervised reference for the missing transition frames.
Approach
Converts unsupervised transition generation into supervised masked-frame prediction by randomly masking segments of long sign videos, and uses a conditional latent diffusion denoiser conditioned on observed frames via cross-attention, initialized at inference with linear interpolation between boundary poses.
Key results
- On PHOENIX14T (20 observed, 10 predicted), Sign-D2C reaches DTW 0.61 and BLEU-1 22.36 on TEST, versus G2P-DDM DTW 12.57 and BLEU-1 9.52 on TEST.
- On USTC-CSL100 (20 observed, 10 predicted), BLEU-1 reaches 69.64 on DEV, close to ground truth 69.10.
- Linear padding gives the lowest DTW (0.61 TEST) among w/o, front, back, and linear padding strategies on PHOENIX14T.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Sign-D2C instead of VQ-GCDM | Sign language transition generation · PHOENIX14T, observation 20 frames - prediction 10 frames | BLEU-1: 9.85 -> 22.36 | Helped |
| Use Sign-D2C instead of VQ-GCDM | Sign language transition generation · PHOENIX14T, observation 10 frames - prediction 20 frames | BLEU-1: 9.02 -> 19.59 | Helped |
| Use Sign-D2C instead of VQ-GCDM | Sign language transition generation · USTC-CSL100, observation 20 frames - prediction 10 frames | BLEU-1: 30.71 -> 69.64 | Helped |
| Use Sign-D2C instead of VQ-GCDM | Sign language transition generation · USTC-CSL100, observation 10 frames - prediction 20 frames | DTW: 10.89 -> 1.79 | Helped |
| Use masking ratio r=0.1 instead of r=0.3 (ablation) | Sign-D2C · PHOENIX14T TEST, observation 20 frames - prediction 10 frames | DTW: 0.61 -> 0.60 | Helped |
| Use masking ratio r=0.1 instead of r=0.3 (ablation) | Sign-D2C · PHOENIX14T TEST, observation 20 frames - prediction 10 frames | BLEU-4: 8.73 -> 8.67 | Hurt |
| Use masking ratio r=0.5 instead of r=0.3 (ablation) | Sign-D2C · PHOENIX14T TEST, observation 20 frames - prediction 10 frames | DTW: 0.61 -> 0.75 | Hurt |
| Use masking ratio r=0.5 instead of r=0.3 (ablation) | Sign-D2C · PHOENIX14T TEST, observation 20 frames - prediction 10 frames | BLEU-4: 8.73 -> 8.35 | Hurt |
| Add linear interpolation padding instead of no padding (ablation) | Sign-D2C · PHOENIX14T DEV, observation 20 frames - prediction 10 frames | DTW: 11.46 -> 0.62 | Helped |
| Use linear interpolation padding instead of front padding (ablation) | Sign-D2C · PHOENIX14T DEV, observation 20 frames - prediction 10 frames | DTW: 0.72 -> 0.62 | Helped |
| Use linear interpolation padding instead of back padding (ablation) | Sign-D2C · PHOENIX14T DEV, observation 20 frames - prediction 10 frames | DTW: 0.67 -> 0.62 | Helped |
Surprising findings
- Lower masking ratio (0.1) gives the best DTW coherence, while higher ratio (0.3) gives slightly better BLEU-4.
Limitations
- No standardized metrics for transition generation; evaluation relies on proxy semantic metrics via NSLT back-translation.
- Baselines were adapted from SLP methods rather than established benchmarks for this task.
- Sign-D2C does not match ground-truth semantic scores on PHOENIX14T (BLEU-1 22.36 vs 29.76).
Experimental setup
Datasets: PHOENIX14T USTC-CSL100 USTC-SLR500
Benchmarks: PHOENIX14T USTC-CSL100 USTC-SLR500
Metrics: BLEU-1 BLEU-4 ROUGE WER DTW FID MPJPE
Baselines: G2P-DDM VQ-GCDM
Builds on: DDPM G2P-DDM VQ-GCDM
Closest papers by meaning
- Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization (NeurIPS 2025 · similarity 0.90)
- Lost in Translation, Found in Context: Sign Language Translation with Contextual Cues (CVPR 2025 · similarity 0.88)
- G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model (AAAI 2024 · similarity 0.88)
- Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation (NeurIPS 2025 · similarity 0.88)
- Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment (AAAI 2024 · similarity 0.87)
- Long-Term TalkingFace Generation via Motion-Prior Conditional Diffusion Model (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.