Research map / Image generation / Autoregressive and latent generation
G2P-DDM: Generating Sign Pose Sequence from Gloss Sequence with Discrete Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes G2P-DDM, a two-stage discrete diffusion approach that maps sign gloss sequences to discrete pose-code sequences (via a multi-codebook Pose-VQVAE), using a CodeUnet denoiser and a sequential-KNN length predictor.
Problem
Gloss-to-pose (G2P) generation for sign language production, where pose sequences have variable length and continuous space generation is hard to model non-autoregressively.
Approach
Discretize pose sequences with a VQ-VAE that uses three local patches (pose, right hand, left hand) each with its own codebook, then model the discrete latent codes with a mask-and-replace discrete diffusion model extended with [PAD] tokens for variable lengths, using a Unet-like fully transformer (CodeUnet) denoiser and a density-peak-based clustering to predict gloss durations.
Key results
- WER improved from 82.01% (prior SOTA NAT-EA) to 77.26% on back-translated gloss generation (5.7% relative).
- BLEU-4 improved from 6.66 to 7.50 (12.6% relative) for back-translated spoken language.
- Deaf user study: 319/350 pairwise preferences for G2P-DDM over the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use G2P-DDM instead of the strongest prior G2P baseline | G2P-DDM · RWTH-PHOENIX-WEATHER-2014T test | WER: 82.01 → 77.26 | Helped |
| Use G2P-DDM instead of the strongest prior G2P baseline | G2P-DDM · RWTH-PHOENIX-WEATHER-2014T test | BLEU-4: 6.66 → 7.50 | Helped |
| Use mask-and-replace diffusion instead of mask-only Mask-Predict (ablation) | G2P-DDM · RWTH-PHOENIX-WEATHER-2014T | back-translation performance | Helped |
| Replace Transformer with CodeUnet (ablation) | G2P-DDM · RWTH-PHOENIX-WEATHER-2014T | back-translation performance | Helped |
| Separate skeleton points into three local patches and use three separate codebooks (ablation) | Pose-VQVAE · RWTH-PHOENIX-WEATHER-2014T | reconstruction MSE | Helped |
| Increase training diffusion steps from 20 to 100 (ablation) | G2P-DDM · RWTH-PHOENIX-WEATHER-2014T | generation performance | Helped |
| Increase training diffusion steps beyond 100 (ablation) | G2P-DDM · RWTH-PHOENIX-WEATHER-2014T | generation performance | No clear effect |
| Increase the number of inference steps (ablation) | G2P-DDM · RWTH-PHOENIX-WEATHER-2014T | generation performance | Helped |
| Use G2P-DDM instead of the baseline method in deaf-user preference evaluation | G2P-DDM · 50 generated pose sequences judged by 7 participants | preferred comparisons: 319/350 preferred our method, while only 31/350 chose the baseline method | Helped |
Surprising findings
- Mask-and-replace diffusion outperforms the mask-only Mask-Predict strategy for G2P.
Limitations
- Evaluation relies on a back-translation pipeline that may introduce noise and is inferior to photo-realistic evaluation.
- Only evaluated on RWTH-PHOENIX-WEATHER-2014T, the single public SLP dataset with parallel data.
- Ground-truth poses are extracted with OpenPose and lifted to 3D, so pose quality is limited by the estimator.
Experimental setup
Datasets: RWTH-PHOENIX-WEATHER-2014T
Benchmarks: RWTH-PHOENIX-WEATHER-2014T
Metrics: WER BLEU-4 MSE
Baselines: Progressive Transformer (PTR) NAT-EA NAT-AT G2P-MP G2P-AR
Builds on: D3PM VQ-Diffusion VQ-VAE Mask-Predict Progressive Transformer
Closest papers by meaning
- Discrete to Continuous: Generating Smooth Transition Poses from Sign Language Observations (CVPR 2025 · similarity 0.88)
- Advanced Sign Language Video Generation with Compressed and Quantized Multi-Condition Tokenization (NeurIPS 2025 · similarity 0.87)
- Bridging Sign and Spoken Languages: Pseudo Gloss Generation for Sign Language Translation (NeurIPS 2025 · similarity 0.87)
- Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Production (AAAI 2025 · similarity 0.87)
- Sign2GPT: Leveraging Large Language Models for Gloss-Free Sign Language Translation (ICLR 2024 · similarity 0.87)
- Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.