Research map / Image generation / Editing and style transfer
Free-viewpoint Human Animation with Pose-correlated Reference Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a pose-correlated, multi-reference diffusion network with adaptive top-K reference selection for free-viewpoint human animation under large viewpoint and camera-distance changes, plus the MSTed TED-video dataset.
Problem
Diffusion-based human animation from a single reference image fails under large viewpoint and camera-distance changes (zoom-in/zoom-out) because single references lack appearance details and pose alignment degrades.
Approach
Uses multiple reference images and a pose correlation module that computes attention maps between non-aligned reference and target poses; an adaptive top-K selection keeps only the most informative reference features, keeping compute roughly constant as references grow.
Key results
- On DyMVHumans, Ours (R=10) reaches L1 35.27, PSNR 35.02, LPIPS 0.1383, FVD 5.459, versus AnimateAnyone at PSNR 33.58 and FVD 12.300.
- With a single reference (R=1) on MSTed, Ours achieves PSNR 32.18 and FVD 20.88, compared with Champ at PSNR 30.87 and FVD 25.68.
- Ablation on MSTed: adding pose correlation and selection to the 2-reference baseline lowers FVD from 26.32 (baseline) to 7.04 (Ours R=2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use two reference images instead of the single-reference baseline (ablation) | Human animation model · MSTed | FVD: 26.32 vs. 9.82 | Helped |
| Add the pose correlation module to the two-reference baseline (ablation) | Human animation model · MSTed | FVD: 9.82 vs. 7.60 | Helped |
| Add reference selection to the two-reference model with pose correlation (ablation) | Human animation model · MSTed | PSNR: 32.20 vs. 32.49 | Helped |
| Increase inference references from one to two | Proposed method · MSTed | PSNR: +0.31 PSNR | Helped |
| Increase inference references from one to two | Proposed method · MSTed | FVD: 20.88 -> 7.044 | Helped |
| Increase inference references from one to ten | Proposed method · DyMVHumans | FVD: 9.047 -> 5.459 | Helped |
| Train with multiple references and use one reference at inference | Proposed method · DyMVHumans | PSNR: 33.58 -> 34.22 | Helped |
| Train with multiple references and use one reference at inference | Proposed method · DyMVHumans | L1: 56.30 -> 56.47 | Hurt |
Surprising findings
- Training with multiple references improves single-reference inference (Ours R=1 beats one-reference baselines on DyMVHumans FVD 9.047 vs 12.300).
Limitations
- Relies on skeleton poses as driving signals without additional modalities.
- Top-K selection uses non-differentiable argsort, requiring a uniform-sampling compensation during training.
- Dataset release is promised only after acceptance; compute details are not stated.
Experimental setup
Datasets: MSTed DyMVHumans
Benchmarks: MSTed DyMVHumans
Metrics: L1 PSNR LPIPS MOVIE FVD
Baselines: MagicAnimate AnimateAnyone Champ
Builds on: AnimateAnyone Champ MagicAnimate temporal aggregation
Closest papers by meaning
- Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (CVPR 2024 · similarity 0.88)
- HumanVid: Demystifying Training Data for Camera-controllable Human Image Animation (NeurIPS 2024 · similarity 0.88)
- DisPose: Disentangling Pose Guidance for Controllable Human Image Animation (ICLR 2025 · similarity 0.87)
- TPC: Test-time Procrustes Calibration for Diffusion-based Human Image Animation (NeurIPS 2024 · similarity 0.87)
- Enhanced Fine-Grained Motion Diffusion for Text-Driven Human Motion Synthesis (AAAI 2024 · similarity 0.86)
- FaceShot: Bring Any Character into Life (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.