Research map / Video / Video generation
DiffPerformer: Iterative Learning of Consistent Latent Guidance for Diffusion-based Human Video Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Finetunes a pose-guided VideoComposer diffusion model on a single target video and iteratively refines its outputs with an implicit video representation (hash-encoded canonical field plus deformation) and 3D-aware human flow loss, feeding the smoothed video back as latent guidance.
Problem
Pose-guided diffusion models for human video generation suffer from temporal inconsistency in appearance and poses due to the stochastic sampling process.
Approach
Use an implicit video representation (CoDef-style canonical field and deformation field with 3D hash encoding) as a temporal proxy that distills diffusion outputs into a smooth video, encode the smoothed video's noisy latents as denoising initialization for the diffusion model, and alternate these steps in a loop; add a 3D-aware human flow from mesh-based correspondences as a temporal constraint.
Key results
- On test poses, DiffPerformer reports FVD 254.39 versus 298.79 for Fast-Vid2Vid, 326.12 for DisCo and 529.13 for DreamPose.
- Image metrics: PSNR 30.72, SSIM 0.69, LPIPS 0.22, FID 36.00, best among compared methods (DisCo FID 55.96).
- Ablations show the implicit video representation removes temporal flicker and 3D-aware human flow corrects limb misalignment under large motion.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add implicit video representation (IVR) optimization with iterative latent guidance | pose-guided diffusion video generation · TikTok / test poses | FVD: FVD 254.39 vs 326.12 for DisCo (best baseline) | Helped |
| Add implicit video representation (IVR) optimization with iterative latent guidance | pose-guided diffusion video generation · TikTok / test poses | FID-VID: 22.32 vs 23.67 for DisCo | Helped |
| Add IVR-based optimization with iterative latent guidance | pose-guided diffusion video generation · TikTok / test poses | PSNR: 30.72 vs 30.4 for Fast-Vid2Vid (best baseline) | Helped |
| Add IVR-based optimization with iterative latent guidance | pose-guided diffusion video generation · TikTok / test poses | LPIPS: 0.22 vs 0.29 for Fast-Vid2Vid (best baseline) | Helped |
| Add 3D-aware human flow loss to implicit video representation (ablation) | IVR optimization · qualitative ablation | pose alignment | Helped |
| Finetune VAE decoder on the target video (ablation) | pose-guided diffusion model · qualitative ablation | photorealistic facial detail | Helped |
| Refine coarse video with diffusion model without IVR (image-level optimization) (ablation) | pose-guided diffusion output · qualitative ablation | temporal consistency | No clear effect |
Surprising findings
- Refining the raw diffusion output with the diffusion model alone does not improve temporal consistency and can introduce color corruption.
Limitations
- Does not work well for videos with a moving camera due to lack of background guidance.
- Diffusion-based pipeline is more time-consuming than non-diffusion methods.
- Requires per-character finetuning on the target video.
Experimental setup
Datasets: TikTok dataset causally captured daily videos
Benchmarks: TikTok dataset daily videos
Metrics: PSNR SSIM LPIPS FID L1 FID-VID FVD
Baselines: Fast-Vid2Vid DreamPose DisCo
Builds on: VideoComposer CoDef Instant-NGP latent diffusion models ControlNet
Closest papers by meaning
- MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance (ICML 2025 · similarity 0.88)
- VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis (CVPR 2025 · similarity 0.87)
- VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control (ICLR 2025 · similarity 0.87)
- Cross-view Masked Diffusion Transformers for Person Image Synthesis (ICML 2024 · similarity 0.87)
- Emergent Temporal Correspondences from Video Diffusion Transformers (NeurIPS 2025 · similarity 0.87)
- Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.