Research map / 3D vision / 3D generation
Controllable Human-centric Keyframe Interpolation with Generative Prior
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PoseFuse3D-KI, a ControlNeXt-style controllable keyframe interpolator built on Wan2.1 that fuses 2D DWPose, SMPL-X renderings, and a novel 3D SMPL-X encoder, and introduces the CHKI-Video dataset for human-centric interpolation.
Problem
Keyframe interpolation between widely separated frames of articulated human motion is ambiguous, and existing video-diffusion interpolators and 2D-skeleton-guided methods produce distorted bodies and offer limited control.
Approach
A PoseFuse3D control model that encodes SMPL-X vertices and joints directly in 3D, projects them to image space via attention aggregation, and fuses them with 2D DWPose and SMPL-X rendering features through coarse-to-fine attention blocks, injected into a video diffusion backbone.
Key results
- On CHKI-Video, PoseFuse3D-KI reaches 22.27 dB PSNR and 0.1304 LPIPS versus 20.42 dB and 0.2100 for FCVG (about +1.85 dB and -0.0796 LPIPS).
- In the in-the-wild setting, PSNR improves 4.5% and LPIPS drops 22.1% relative to FCVG.
- A user study with 40 participants preferred PoseFuse3D-KI in 86.5% of comparisons.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 3D-informed control with PoseFuse3D-KI instead of FCVG | CHKI-Video keyframe interpolation · CHKI-Video test | PSNR: 20.42 -> 22.27; +1.85 dB | Helped |
| add 3D-informed control with PoseFuse3D-KI instead of FCVG | CHKI-Video keyframe interpolation · CHKI-Video test | LPIPS: 0.2100 -> 0.1304; -0.0796 | Helped |
| add 3D control to the 2D-only FCVG strategy | VE-SVD · CHKI-Video test | PSNR: 20.42 -> 20.96; more than 0.50 dB on all the PSNRs | Helped |
| add direct SMPL-X encoding to visual encoding (VE) (ablation) | PoseFuse3D · CHKI-Video test | LPIPS_bbox: 0.0682 -> 0.0653 | Helped |
| add depth and normal renderings to VE (ablation) | PoseFuse3D · CHKI-Video test | LPIPS: 0.1400 -> 0.1363 | Helped |
| add 3D visual encoding (ablation) | PoseFuse3D · CHKI-Video test, 3× temporally downsampled | PSNR_bbox: 16.94 -> 17.30; 0.26 dB | Helped |
| use the designed fusion module instead of summation (ablation) | PoseFuse3D · CHKI-Video test, 3× temporally downsampled | LPIPS_mask: 0.0806 -> 0.0773; increase of 0.0033 when replaced by Sum | Helped |
| use shifted window partitioning in cross-attention (ablation) | PoseFuse3D · CHKI-Video test, 3× temporally downsampled | PSNR_bbox: 0.11 dB | Helped |
| include temporal self-attention in fusion (ablation) | PoseFuse3D · CHKI-Video test, 3× temporally downsampled | LPIPS_bbox: 0.0022 | Helped |
| include visual encodings of SMPL-X and DWPose (ablation) | PoseFuse3D · CHKI-Video test | PSNR_mask: 4.32 dB | Helped |
| include 2D DWPose visualization encoding (ablation) | PoseFuse3D · CHKI-Video test | LPIPS_bbox: 13% | Helped |
| include joint aggregation in the SMPL-X encoder (ablation) | PoseFuse3D · CHKI-Video test | PSNR_bbox: 0.06 dB | Helped |
Surprising findings
- Control-free methods such as GI and Wan2.1-KI sometimes generate undistorted humans, and methods with fewer learned priors (GIMM-VFI) achieve the highest HA score on the easier FCVG-Test-HC benchmark.
Limitations
- Depends on accurate SMPL-X estimation; estimator errors degrade interpolation.
- Inherits the base diffusion model's priors and high GPU memory demands.
- Does not explicitly model human-object interactions.
Experimental setup
Datasets: CHKI-Video SportsSlomo Pexels
Benchmarks: CHKI-Video FCVG-Test-HC
Metrics: PSNR LPIPS PSNR_bbox PSNR_mask LPIPS_bbox LPIPS_mask HA user preference
Baselines: FCVG GI GIMM-VFI Wan2.1-KI SVD (VE-SVD variant)
Builds on: ControlNeXt Wan2.1 FCVG Champ SMPLer-X
Closest papers by meaning
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.86)
- DisPose: Disentangling Pose Guidance for Controllable Human Image Animation (ICLR 2025 · similarity 0.85)
- HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation (NeurIPS 2024 · similarity 0.85)
- 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation (ICLR 2025 · similarity 0.85)
- VideoMDM: Towards 3D Human Motion Generation From 2D Supervision (NeurIPS 2026 · similarity 0.85)
- ONE-SHOT: Compositional Human-Environment Video Synthesis via Spatial-Decoupled Motion Injection and Hybrid Context Integration (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.