Research map / 3D vision / 3D generation
RePerformer: Immersive Human-centric Volumetric Videos from Playback to Photoreal Reperformance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RePerformer, a 3D Gaussian-based representation that disentangles motion and appearance Gaussians, packs appearance Gaussians into Morton-ordered 2D maps regressed by U-Nets, and transfers motion for photoreal re-performance of human-centric volumetric videos.
Problem
Playback methods for human-centric volumetric videos reconstruct dynamic scenes faithfully but cannot generalize to novel motions, while animatable avatar methods are limited to human-only scenes and need extensive motion data; general human-object scenes need both playback and re-performance.
Approach
Hierarchically disentangle a dynamic scene into sparse motion Gaussians (driving warping) and dense appearance Gaussians, map appearance Gaussian positions to 2D maps via Morton ordering so 2D U-Nets (with self-attention) regress attributes generalizably, and re-perform by semantic-aware alignment plus as-rigid-as-possible deformation transfer of motion Gaussians.
Key results
- On the DualGS dataset novel view synthesis, RePerformer reaches 34.571 PSNR / 0.986 SSIM / 0.0229 LPIPS at 1.68 min/frame training, ranking second behind DualGS (35.506 PSNR) while training much faster.
- On a 3,000-frame generalization test, RePerformer achieves 32.09 PSNR on novel views and 30.06 PSNR on novel motions, outperforming AP-NeRF, TAVA, and SC-GS.
- Morton-based parameterization converges faster and renders better than y-axis or random 3D-to-2D mappings; the three-U-Net design improves PSNR by 0.25 dB over a single network.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RePerformer instead of DualGS | DualGS dataset novel-view rendering · DualGS dataset, novel view | PSNR: 35.506 -> 34.571 | Hurt |
| Use RePerformer instead of DualGS | DualGS dataset novel-view rendering · DualGS dataset, novel view | SSIM: 0.990 -> 0.986 | Hurt |
| Use RePerformer instead of V3 | DualGS dataset novel-view rendering · DualGS dataset, novel view | LPIPS: 0.0167 -> 0.0229 | Hurt |
| Use RePerformer instead of V3 | DualGS dataset novel-view rendering · DualGS dataset, novel view | Training Time (Min / frame): 1.86 -> 1.68 | Helped |
| Use RePerformer instead of SC-GS | template-free modeling approaches · Novel view | PSNR: 30.63 -> 32.09 | Helped |
| Use RePerformer instead of SC-GS | template-free modeling approaches · Novel motion | LPIPS: 0.0821 -> 0.0277 | Helped |
| Use Morton parameterization instead of y-axis order or random mapping (ablation) | RePerformer training strategy · 500-frame ablation | Rendering quality | Helped |
| Add the warm-up stage (ablation) | RePerformer training strategy · 500-frame ablation | Convergence speed | Helped |
| Use three specialized U-Nets instead of a single network for all attributes (ablation) | RePerformer on the DualGS dataset · DualGS dataset | PSNR: +0.25 dB | Helped |
| Add semantic alignment loss (ablation) | RePerformer alignment · Re-performance alignment | Alignment quality | Helped |
| Add ARAP loss to alignment (ablation) | RePerformer alignment · Re-performance alignment | Topology consistency | Helped |
| Add ARAP loss to motion transfer (ablation) | RePerformer motion transfer · Novel motion re-performance | Rendering quality | Helped |
Surprising findings
- Per-frame optimized DualGS still has the best novel-view PSNR, so RePerformer trades a small quality drop for speed and generalization.
Limitations
- Motion Gaussian tracking takes about 20 seconds per frame, limiting real-time use.
- Generalization to arbitrary motions is limited; re-performance relies on similar interaction types in training data.
- Only one self-attention layer was used due to GPU memory constraints.
Experimental setup
Datasets: DualGS dataset
Benchmarks: DualGS dataset
Metrics: PSNR SSIM LPIPS training time per frame
Baselines: NeuS2 Spacetime Gaussian DualGS V3 Articulated-Point-NeRF TAVA SC-GS
Builds on: 3D Gaussian Splatting DualGS Dynamic 3D Gaussians Embedded deformation graph deformation transfer
Closest papers by meaning
- GaussianAvatar: Towards Realistic Human Avatar Modeling from a Single Video via Animatable 3D Gaussians (CVPR 2024 · similarity 0.85)
- ODHSR: Online Dense 3D Reconstruction of Humans and Scenes from Monocular Videos (CVPR 2025 · similarity 0.84)
- HiFi4G: High-Fidelity Human Performance Rendering via Compact Gaussian Splatting (CVPR 2024 · similarity 0.84)
- LIFe-GoM: Generalizable Human Rendering with Learned Iterative Feedback Over Multi-Resolution Gaussians-on-Mesh (ICLR 2025 · similarity 0.84)
- OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation (CVPR 2025 · similarity 0.84)
- GASP: Gaussian Avatars with Synthetic Priors (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.