Research map / 3D vision / 3D generation
From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates photorealistic full-body conversational avatars (face, body, hands) from dyadic speech audio by combining a VQ transformer for guide poses with a diffusion model that infills high-frequency motion, plus a new multi-view conversational dataset.
Problem
Generating diverse, realistic face, body, and hand gestures for an individual in a dyadic conversation from speech audio, and evaluating such gestures faithfully, since mesh/skeleton renderings hide subtle nuances.
Approach
Combine an autoregressive residual-VQ transformer that produces diverse coarse guide poses at 1fps with an audio- and guide-conditioned diffusion model that infills high-frequency body/hand motion at 30fps; face is generated by a separate audio- and lip-conditioned diffusion model; motion is rendered on personalized photorealistic avatars.
Key results
- Achieves lowest geometric Fréchet distance FDg of 2.94 vs 4.97 for SHOW and 5.08 for LDA, with highest pose diversity Divg 2.98 (GT 3.09).
- Mechanical Turk A/B tests show about 70% of evaluators prefer the method over LDA in both mesh and photoreal settings.
- In photoreal A/B tests vs ground truth, 43% of evaluators strongly prefer ground truth, indicating photorealism reveals gesture differences that meshes hide.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine guide-pose conditioning with audio conditioning in pose diffusion (ablation) | Ours body motion model · Conversation test set | FDg: 3.94 -> 2.94 | Helped |
| Add guide-pose conditioning to audio-conditioned motion diffusion (ablation) | Ours body motion model · Conversation test set | Divsample: 2.06 -> 3.58 | Helped |
| Add audio conditioning to guide-pose-conditioned pose diffusion (ablation) | Ours body motion model · Conversation test set | FDg: 3.94 -> 2.94 | Helped |
| Add the pretrained lip regressor as conditioning (ablation) | Ours face motion model · Conversation test set | Horizontal L2 Error: 2.62 -> 2.29 | Helped |
| Use the proposed VQ-guide-pose plus diffusion method rather than SHOW | Conversational motion generation · Conversation test set | FDg: 4.97 -> 2.94 | Helped |
| Use the proposed method rather than LDA | Conversational motion generation · Conversation test set | Divsample: 2.68 -> 3.58 | Helped |
Surprising findings
- Evaluators were more forgiving of incorrect motion on meshes, and distinguished ground truth from generated motion more often with photorealistic rendering than with meshes.
Limitations
- Operates on short-range audio and cannot generate gestures needing long-range language understanding (e.g. counting).
- Photorealistic generation limited to four subjects in the dataset, for consent reasons.
- Ground-truth-level realism lags in photoreal perceptual tests vs ground truth.
Experimental setup
Datasets: Panoptic Studio triadic haggling data (Joo et al.) referenced for context in-house multi-view dyadic conversation dataset
Benchmarks: in-house multi-view conversational dataset
Metrics: FDg FDk Divg Divk Divsample lip L2 error pairwise preference
Baselines: Random KNN SHOW LDA Ours Uncond Ours w/o P Ours w/o A
Builds on: DDPM residual VQ-VAE LDA (latent diffusion audio-to-gesture) SHOW Codec Avatars
Closest papers by meaning
- VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time (NeurIPS 2024 · similarity 0.88)
- The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion (CVPR 2025 · similarity 0.88)
- Towards Variable and Coordinated Holistic Co-Speech Motion Generation (CVPR 2024 · similarity 0.87)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.87)
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.87)
- Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.