Research map / 3D vision / Face and shape
PTUS: Photo-Realistic Talking Upper-Body Synthesis via 3D-Aware Motion Decomposition Warping
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PTUS, a 3D-aware motion-decomposition warping framework that animates both upper body and face (blinking, lip sync) from a source image and driving video, and releases the Talking-UB dataset.
Problem
Talking upper-body synthesis: animating body and face of a source image from a driving video while preserving precise facial details, which prior head-only or body-only methods fail to do jointly.
Approach
Decomposes 3D keypoint motion into separate face and upper-body motions (face-body motion decomposition) and splits the face motion into global and local parts (local-global decomposition), then applies coarse-to-fine 3D-aware warping of depth-aware features, extending OSFV.
Key results
- PTUS achieves L1 0.035 and FLD 1.322 on Talking-UB same-identity transfer, the best among FOMM, MRAA, LIA, TPSMM (best baseline TPSMM L1 0.036, FLD 1.377).
- Cross-identity video FID of 26.61 for PTUS vs 34.28 for TPSMM and 82.63 for LIA.
- Human study (40 raters) prefers PTUS in 64.00% of identity judgments and 50.50% of authenticity judgments, versus 23.25% for the next best (TPSMM) on authenticity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PTUS instead of the strongest baseline TPSMM | Talking-UB same-identity motion transfer · Talking-UB same-identity | L1: 0.036 -> 0.035 | Helped |
| use PTUS instead of the strongest baseline TPSMM | Talking-UB same-identity motion transfer · Talking-UB same-identity | AKD: 2.46 -> 2.46 | No clear effect |
| use PTUS instead of the strongest baseline TPSMM | Talking-UB same-identity motion transfer · Talking-UB same-identity | AED: 0.043 -> 0.043 | No clear effect |
| use PTUS instead of the strongest baseline TPSMM | Talking-UB same-identity motion transfer · Talking-UB same-identity | FLD: 1.377 -> 1.322 | Helped |
| use PTUS instead of the strongest baseline TPSMM | Talking-UB cross-identity motion transfer · Talking-UB cross-identity | FID: 34.28 -> 26.61 | Helped |
| add face-body motion decomposition (separate face and body motion estimation) (ablation) | PTUS · Talking-UB | L1: 0.068 -> 0.035 | Helped |
| add face-body motion decomposition (separate face and body motion estimation) (ablation) | PTUS · Talking-UB | AKD: 9.85 -> 2.46 | Helped |
| add face-body motion decomposition (separate face and body motion estimation) (ablation) | PTUS · Talking-UB | FID: 86.44 -> 26.61 | Helped |
| add local-global motion decomposition of facial motion (ablation) | PTUS · Talking-UB | L1: 0.067 -> 0.035 | Helped |
| add local-global motion decomposition of facial motion (ablation) | PTUS · Talking-UB | AKD: 10.12 -> 2.46 | Helped |
| add local-global motion decomposition of facial motion (ablation) | PTUS · Talking-UB | FID: 87.26 -> 26.61 | Helped |
Surprising findings
- LIA performs much worse than the other methods on same-identity metrics (L1 0.141, FID 82.63) in the paper's comparison.
Limitations
- Upper-body parts such as clothing can blend into the background, requiring a segmentation and background-inpainting post-process at inference
- Evaluated only on the authors' Talking-UB dataset at 256x256 resolution
Experimental setup
Datasets: Talking-UB TED-Talk
Benchmarks: Talking-UB
Metrics: L1 AKD AED FLD FID human preference
Baselines: FOMM MRAA LIA TPSMM OSFV
Builds on: OSFV FOMM MRAA TPSMM
Closest papers by meaning
- Synergizing Motion and Appearance: Multi-Scale Compensatory Codebooks for Talking Head Video Generation (CVPR 2025 · similarity 0.88)
- Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis (ICLR 2024 · similarity 0.87)
- MimicTalk: Mimicking a personalized and expressive 3D talking face in minutes (NeurIPS 2024 · similarity 0.87)
- Teller: Real-Time Streaming Audio-Driven Portrait Animation with Autoregressive Motion Generation (CVPR 2025 · similarity 0.87)
- Occlusion-Insensitive Talking Head Video Generation via Facelet Compensation (AAAI 2025 · similarity 0.87)
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.