Research map / Video / Video generation
Co-speech Gesture Video Generation with 3D Human Meshes
Aniruddha Mahapatra, Richa Mishra, Ziyi Chen, Boyang Ding, Renda Li, Shoulei Wang, Jun-Yan Zhu, Peng Chang, Mei Han, Jing Xiao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use textured SMPL-X meshes as conditioning instead of 2D keypoints | GAN video generator · TalkSHOW Oliver | LPIPS: 0.482 -> 0.408 | Improved |
| add textured meshes as conditioning (ablation) | Mesh-grounded video generator · TalkSHOW Seth | LPIPS: 0.328 -> 0.312 | Improved |
| add textured meshes as conditioning (ablation) | Mesh-grounded video generator · User study, Chemistry | % users preferring Ours: 88% | Improved |
| add depth and normal maps as conditioning (ablation) | Mesh-grounded video generator · TalkSHOW Oliver | LPIPS: 0.407 -> 0.408 | No clear effect |
| add depth and normal maps as conditioning (ablation) | Mesh-grounded video generator · User study, Chemistry | % users preferring Ours: 42% prefer Ours on Chemistry; 66% on Seth | Mixed |
| use a larger UV texture map (512 instead of 256) (ablation) | Mesh texturing · TalkSHOW | visual quality | No clear effect |
| use per-vertex textures instead of UV texture map (ablation) | Mesh texturing · TalkSHOW | visual quality | No clear effect |
| use dynamic NeRF (INST) instead of textured mesh representation (ablation) | Appearance representation · TalkSHOW | hand artifacts | Hurt |
Closest papers by meaning
- Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model (CVPR 2024 · similarity 0.89)
- Co-Speech Gesture Video Generation with Implicit Motion-Audio Entanglement (CVPR 2025 · similarity 0.89)
- Co$^{\mathbf{3}}$Gesture: Towards Coherent Concurrent Co-speech 3D Gesture Generation with Interactive Diffusion (ICLR 2025 · similarity 0.89)
- DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation (CVPR 2024 · similarity 0.87)
- AudCast: Audio-Driven Human Video Generation by Cascaded Diffusion Transformers (CVPR 2025 · similarity 0.87)
- DIDiffGes: Decoupled Semi-Implicit Diffusion Models for Real-time Gesture Generation from Speech (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
