Research map / Video / Video generation
Neural Sign Actors: A Diffusion Model for 3D Sign Language Production from Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a 3D ASL dataset by fitting SMPL-X to How2Sign and trains a text-conditioned autoregressive diffusion model with an anatomically informed GNN to generate 3D signing avatars directly from text.
Problem
Sign language production from text has relied on 2D joints or glosses, producing unrealistic motion; no large-scale 3D sign language data with SMPL-X annotations existed.
Approach
A diffusion model over SMPL-X pose and expression parameters, with a kinematic-tree GNN pose encoder (plus joint-index embeddings to break permutation equivariance), CLIP sentence conditioning, and an LSTM decoder, trained on 3D annotations of How2Sign obtained via an OSX-initialized, Mediapipe-refined, PCA-prior fitting pipeline.
Key results
- On How2Sign, the proposed method reaches MPVPE 31.47 and FID 1.56 for the body, versus 55.02 and 4.96 for the best prior baseline (Stoll et al.).
- Back-translation BLEU-4 of 13.12 vs 7.51 for Stoll et al.
- In a 15-subject ASL user study, the proposed method scored 5.8 alignment vs 8.1 for 3D fittings and 8.7 for ground-truth video; top-1 accuracy 40% and top-2 80% on sentence ranking.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Neural Sign Actors instead of prior SLP methods | 3D sign production · How2Sign test | Body MPVPE (mm): 55.02 -> 31.47 | Helped |
| use Neural Sign Actors instead of prior SLP methods | 3D sign production · How2Sign test | BLEU-4: 7.51 -> 13.12 | Helped |
| add the anatomically informed GNN pose encoder (ablation) | Neural Sign Actors · How2Sign test | MPVPE: 37.51 -> 31.47 | Helped |
| add pose and expression embeddings (ablation) | Neural Sign Actors · How2Sign test | MPVPE: 66.56 -> 31.47 | Helped |
| add the LSTM decoder (ablation) | Neural Sign Actors · How2Sign test | DTW: 10.82 -> 7.83 | Helped |
| replace the LSTM decoder with a Transformer (ablation) | Neural Sign Actors · How2Sign test | MPVPE: 31.47 -> 32.73 | Hurt |
| use CLIP sentence encoding instead of trainable word embeddings (ablation) | Neural Sign Actors · How2Sign test | MPVPE: 69.12 -> 31.47 | Helped |
| use CLIP sentence encoding instead of DistilBERT (ablation) | Neural Sign Actors · How2Sign test | MPVPE: 45.32 -> 31.47 | Helped |
Surprising findings
- The LSTM decoder matches a Transformer decoder in performance while using 75% fewer parameters (1M vs 4.5M).
Limitations
- Baselines were retrained on RWTH-Phoenix rather than How2Sign, and the dataset is ASL-only while prior methods target German SL.
- User study shows generated signs still rated well below ground truth (5.8 vs 8.7 alignment).
Experimental setup
Datasets: How2Sign RWTH-PHOENIX-Weather 2014T SGNify mocap AMASS MANO hand pose dataset
Benchmarks: How2Sign test set SGNify mocap
Metrics: MPVPE MPJPE FID DTW BLEU ROUGE alignment score
Baselines: Saunders et al. (Progressive transformers) Saunders et al. (adversarial SLP) Hwang et al. Stoll et al. (SMPL-X fitting) MDM MotionDiffuse
Builds on: OSX Human Motion Diffusion Model (MDM) DDPM SGNify
Closest papers by meaning
- KD-MSLRT: Lightweight Sign Language Recognition Model Based on Mediapipe and 3D to 1D Knowledge Distillation (AAAI 2025 · similarity 0.89)
- Sign-IDD: Iconicity Disentangled Diffusion for Sign Language Production (AAAI 2025 · similarity 0.87)
- Uni-Sign: Toward Unified Sign Language Understanding at Scale (ICLR 2025 · similarity 0.87)
- Geo-Sign: Hyperbolic Contrastive Regularisation for Geometrically Aware Sign Language Translation (NeurIPS 2025 · similarity 0.86)
- VSNet: Focusing on the Linguistic Characteristics of Sign Language (CVPR 2025 · similarity 0.86)
- Scaling Sign Language Translation (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.