Research map / 3D vision / 3D generation

IM-Portrait: Learning 3D-aware Video Diffusion for Photorealistic Talking Heads from Monocular VideosC

CVPR 2025 · 3D vision › 3D generation · approach: Diffusion-based novel view synthesis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes IM-Portrait, a 3D-aware video diffusion model that directly generates multiplane-image talking-head videos from one identity image and expression/pose signals, trained only on monocular videos via reference-target alternating training.

Problem

Generating photorealistic, 3D-consistent talking head videos renderable from novel viewpoints (e.g., stereo for VR) without multi-view training data or a separate 3D reconstruction stage.

Approach

Diffusion denoises directly in MPI (multiplane image) space with differentiable MPI rendering inside the denoising loop; an alternating training scheme builds MPIs either in the target camera (ground-truth supervision) or the reference camera (bootstrapped pseudo-ground-truth supervision for side views), plus late-stage noise sampling to tolerate pseudo-GT mismatch.

Key results

• On HDTF, the method achieves FVD 69.04 vs 117.54 for Follow-your-emoji and 132.90 for X-Portrait (best video quality), with L1 0.031 and LPIPS 0.118.
• Renders at 109 FPS at 512x512 on a single A100, versus 10 FPS for Portrait4D-v2 (NeRF-based) and 0.1-0.5 FPS for the diffusion baselines.
• Ablation: removing late-stage noise sampling raises Talkinghead1kh FID from 27.83 to 29.86 and degrades fine details such as wrinkles and hair strands.

Measured effects reported in the paper

Surprising findings

• Image-quality metrics (L1, LPIPS, FID) are mixed: the model is not always best, and the w/o-residual variant scores better on several frontal metrics, though it yields worse geometry.

Limitations

• Rendering from excessive camera viewpoint changes is not supported, a known limitation of the MPI representation.
• Training relies on a self-collected, non-public 35k-video dataset and on bootstrapped pseudo ground truth, which may not match the real image distribution.

Experimental setup

Datasets: self-collected 35k monocular talking head videos HDTF Talkinghead1kh

Benchmarks: HDTF Talkinghead1kh

Metrics: L1 LPIPS FID FVD render speed (FPS)

Baselines: Face-V2V EmoPortrait Portrait4D-v2 Follow-your-emoji X-Portrait

Builds on: Lumiere MPI rendering (Tucker and Snavely) Single-view MPI / diffusion-based 3D generation (Tewari et al.) DDPM classifier-free guidance

Closest papers by meaning

• GAF: Gaussian Avatar Reconstruction from Monocular Videos via Multi-view Diffusion (CVPR 2025 · similarity 0.89)
• DiffusionAvatars: Deferred Diffusion for High-fidelity 3D Head Avatars (CVPR 2024 · similarity 0.88)
• VASA-3D: Lifelike Audio-Driven Gaussian Head Avatars from a Single Image (NeurIPS 2025 · similarity 0.88)
• CAP4D: Creating Animatable 4D Portrait Avatars with Morphable Multi-View Diffusion Models (CVPR 2025 · similarity 0.88)
• High-Fidelity 3D Head Avatars Reconstruction through Spatially-Varying Expression Conditioned Neural Radiance Field (AAAI 2024 · similarity 0.88)
• Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis (ICLR 2024 · similarity 0.88)