Research map / 3D vision / NeRF and radiance fields
Real-Time Neural Radiance Talking Portrait Synthesis via Audio-Spatial Decomposition
Jiaxiang Tang, Kaisiyuan Wang, Hang Zhou, Xiaokang Chen, Dongliang He, Tianshu Hu, Jingtuo Liu, Ziwei Liu, Gang Zeng, Jingdong Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use decomposed audio-spatial modulation instead of deformation backbone (ablation) | Ours (head model) · Self-driven (Obama) | LMD: 3.225 -> 2.696 | Improved |
| use decomposed audio-spatial modulation instead of deformation backbone (ablation) | Ours (head model) · Self-driven (Obama) | PSNR: 33.43 -> 34.07 | Improved |
| use decomposed grids instead of composed 5D audio-spatial grid (ablation) | Ours (head model) · Self-driven (Obama) | FPS: 35 -> 40 | Improved |
| use decomposed modulation instead of implicit MLP backbone (ablation) | Ours (head model) · Self-driven (Obama) | FPS: 28 -> 40 | Improved |
| use decomposed modulation instead of implicit MLP backbone (ablation) | Ours (head model) · Self-driven (Obama) | LPIPS: 0.0324 -> 0.0375 | Hurt |
| raise audio coordinate dimension from 2 to 3 (ablation) | Ours · Self-driven (Obama) | FPS / LMD: 40 -> 36 FPS; 2.696 -> 2.749 LMD | Hurt |
| raise audio coordinate dimension from 1 to 2 (ablation) | Ours · Self-driven (Obama) | LMD: 2.736 -> 2.696 | Improved |
| increase ray sampling count from 8 to 16 (ablation) | Ours · Self-driven (Obama) | FPS / LMD: 51 -> 40 FPS; 2.761 -> 2.696 LMD | Mixed |
| increase ray sampling count from 16 to 32 (ablation) | Ours · Self-driven (Obama) | LMD / FPS: 2.696 -> 2.661 LMD; 40 -> 35 FPS | No clear effect |
| use proposed method instead of AD-NeRF | Ours · Self-driven (Obama) | PSNR: 25.53 -> 34.00 | Improved |
| use proposed method instead of Wav2Lip for lip sync | Ours · Cross-driven Testset A | Sync: 9.257 -> 6.218 | Hurt |
Closest papers by meaning
- AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head Synthesis (AAAI 2024 · similarity 0.91)
- Real-time 3D-aware Portrait Video Relighting (CVPR 2024 · similarity 0.91)
- GoHD: Gaze-oriented and Highly Disentangled Portrait Animation with Rhythmic Poses and Realistic Expressions (AAAI 2025 · similarity 0.88)
- Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis (ICLR 2024 · similarity 0.88)
- SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis (CVPR 2024 · similarity 0.88)
- DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking head Video Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
