Research map / 3D vision / Face and shape
3D²-Actor: Learning Pose-Conditioned 3D-Aware Denoiser for Realistic Gaussian Avatar Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes 3D²-Actor, a pose-conditioned 3D-aware denoiser that interleaves a pose-conditioned 2D diffusion denoiser with a Gaussian-based 3D rectifier to reconstruct and animate human avatars from multi-view RGB video.
Problem
Animatable 3D human avatar modeling from sparse multi-view RGB videos struggles to capture pose-dependent clothing details and generalize to novel poses, and 2D diffusion models lack 3D/temporal consistency for human-centric video.
Approach
Interleave pose-conditioned (SMPL segmentation via SFT layers) 2D denoising steps with 3D rectifying steps that reconstruct 3D Gaussians from clean anchor-view images; Gaussians are parameterized by mesh-based local coordinates (barycentric plus offset) with two-stage projection, and inter-frame consistency is enforced by propagating Gaussian local coordinates from the previous frame during sampli
Key results
- On four ZJU-MoCap sequences (313, 315, 377, 386), the method achieves best LPIPS and FID on most sequences, e.g., sequence 313 FID 19.5 vs 27.5 for PoseVocab and 32.6 for ARAH, LPIPS 0.080 vs 0.101.
- ARAH achieves the highest PSNR on all four sequences, while the proposed method has balanced PSNR (23.5 on 313).
- Ablation shows 2D-only denoising has better LPIPS/FID than 3D-only (313 FID 16.8 vs 115.8), and the full combination gives 19.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 3D²-Actor instead of ARAH or PoseVocab | Novel-pose avatar synthesis · ZJU-MoCap sequences 313, 315, 377, 386 | LPIPS and FID: LPIPS: 0.097 -> 0.080; FID: 32.6 -> 19.5 | Helped |
| Use 3D²-Actor instead of ARAH or PoseVocab | Novel-pose avatar synthesis · ZJU-MoCap sequences 313, 315, 377, 386 | PSNR: 313: 24.3 -> 23.5; 315: 20.9 -> 20.7; 377: 24.8 -> 24.4; 386: 27.9 -> 26.8 | Mixed |
| Add the 3D rectifier to 2D-only denoising (ablation) | 3D²-Actor · ZJU-MoCap 313 and 315 novel pose | LPIPS: 313: 0.086 -> 0.080; 315: 0.098 -> 0.090 | Helped |
| Add the 2D denoiser to the initial 3D rectifier (ablation) | 3D²-Actor · ZJU-MoCap 313 and 315 novel pose | LPIPS: 313: 0.148 -> 0.080; 315: 0.150 -> 0.090 | Helped |
| Add the 2D denoiser to the initial 3D rectifier (ablation) | 3D²-Actor · ZJU-MoCap 313 and 315 novel pose | FID: 313: 115.8 -> 19.5; 315: 79.7 -> 20.2 | Helped |
| Increase t_split from 200 to 300 (ablation) | 3D²-Actor 3D-aware denoising · ZJU-MoCap sequences 313, 315, 377, 386 | LPIPS and FID: 313 LPIPS: 0.082 -> 0.080; FID: 21.5 -> 19.5 | Helped |
| Vary 3D rectifier count k from 2 to 4 at fixed t_split (ablation) | 3D²-Actor 3D-aware denoising · ZJU-MoCap sequences 313, 315, 377, 386 | LPIPS and FID | No clear effect |
| Add mesh-based Gaussian local coordinates (ablation) | 3D²-Actor · Novel-pose avatar synthesis | visual quality | Helped |
| Use Gaussian consistency sampling instead of independent per-frame noise sampling (ablation) | 3D²-Actor video synthesis · Consecutive novel-pose frames | inter-frame continuity | Helped |
Surprising findings
- The 3D-only rectifier path yields much worse FID (115.8 on sequence 313) than 2D-only, and the full interleaved process is worse than 2D-only on FID for sequence 313 (19.5 vs 16.8).
- PSNR is reported as less meaningful for novel poses because clothing stochasticity makes ground truth only one possible outcome, and ARAH has the highest PSNR despite worse perceptual metrics.
Limitations
- Metrics like PSNR may not capture fidelity for novel poses due to clothing stochasticity; relies on LPIPS and FID.
- Hyperparameters t_split and k must be searched per identity, and evaluation is limited to four ZJU-MoCap sequences.
- Only single-actor, sparse multi-view ZJU-MoCap setting evaluated; comparisons limited to two baselines.
Experimental setup
Datasets: ZJU-MoCap
Benchmarks: ZJU-MoCap
Metrics: PSNR LPIPS FID
Baselines: ARAH PoseVocab
Builds on: 3D Gaussian Splatting DDIM SFTGAN RenderDiffusion Neural Body
Closest papers by meaning
- Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models (NeurIPS 2024 · similarity 0.90)
- AniGS: Animatable Gaussian Avatar from a Single Image with Inconsistent Gaussian Reconstruction (CVPR 2025 · similarity 0.89)
- 3DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splatting (CVPR 2024 · similarity 0.89)
- Vid2Avatar-Pro: Authentic Avatar from Videos in the Wild via Universal Prior (CVPR 2025 · similarity 0.88)
- GAF: Gaussian Avatar Reconstruction from Monocular Videos via Multi-view Diffusion (CVPR 2025 · similarity 0.88)
- DiffHuman: Probabilistic Photorealistic 3D Reconstruction of Humans (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.