Research map / Video / Video generation
Large Displacement Motion Transfer with Unsupervised Anytime Interpolation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an unsupervised keypoint-based anytime interpolation method that inserts intermediate frames between source and driving images, selects an optimal interpolant, and uses bidirectional training with ViT-based constraints to make large-displacement motion transfer easier.
Problem
Unsupervised motion transfer struggles to accurately model large displacement motion when the pose difference between source and driving images is large.
Approach
Decompose a large displacement into smaller ones by generating unsupervised interpolated images at intermediate times (linear keypoint interpolation), select the best interpolant with an appearance-plus-keypoint-distance selector, and train with a bidirectional scheme plus ViT appearance and structure consistency losses since no paired interpolation labels exist.
Key results
- On Tai-Chi-HD, AKD and MKR improve over TPSMM by 7.88% and 22.2% in video reconstruction.
- On Fashion, AKD and MKR improve over TPSMM by 5.3% and 20%.
- The Ours-V2 multi-view extension raises AKD gains of 12.8% on Tai-Chi-HD, 3.6% on Fashion and 26.1% on UvA-Nemo over the single-view model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add unsupervised anytime interpolation to TPSMM (ablation) | TPSMM · Tai-Chi-HD | AKD: 4.57 -> 5.56 | Hurt |
| Add unsupervised anytime interpolation to TPSMM (ablation) | TPSMM · Tai-Chi-HD | MKR: 0.018 -> 0.026 | Hurt |
| Add bidirectional training to Model1 (ablation) | Model1 · Tai-Chi-HD | AKD: 5.56 -> 4.31 | Helped |
| Add bidirectional training to Model1 (ablation) | Model1 · Tai-Chi-HD | MKR: 0.026 -> 0.016 | Helped |
| Add appearance and structural consistency losses to Model2 (ablation) | Model2 · Tai-Chi-HD | AKD: 4.31 -> 4.21 | Helped |
| Add appearance and structural consistency losses to Model2 (ablation) | Model2 · Tai-Chi-HD | MKR: 0.016 -> 0.014 | Helped |
| Fuse source-image and optimal-interpolation features | Ours · Tai-Chi-HD | AKD: 4.21 -> 3.67 | Helped |
| Fuse source-image and optimal-interpolation features | Ours · Fashion | AKD: 0.800 -> 0.771 | Helped |
| Fuse source-image and optimal-interpolation features | Ours · UvA-Nemo | AKD: 1.155 -> 0.853 | Helped |
Surprising findings
- Naively adding the anytime interpolation module to TPSMM worsens performance (L1 0.045 to 0.053), and only the bidirectional constraint makes it helpful.
Limitations
- Interpolated images are blurred and lose appearance detail, and occlusion regions expand on datasets with background.
- L1 and AED are slightly worse than existing methods, and clothing color can drift from the source image.
Experimental setup
Datasets: UvA-Nemo Fashion Tai-Chi-HD TedTalks
Benchmarks: Tai-Chi-HD Fashion UvA-Nemo TedTalks
Metrics: L1 AKD MKR AED
Baselines: X2Face FOMM MRAA DAM MTIA TPSMM CPABMM
Builds on: FOMM TPSMM MRAA CPABMM DINO ViT
Closest papers by meaning
- MotionStone: Decoupled Motion Intensity Modulation with Diffusion Transformer for Image-to-Video Generation (CVPR 2025 · similarity 0.84)
- DrivingForward: Feed-forward 3D Gaussian Splatting for Driving Scene Reconstruction from Flexible Surround-view Input (AAAI 2025 · similarity 0.83)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.83)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.83)
- AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers (CVPR 2025 · similarity 0.83)
- Continuous Piecewise-Affine Based Motion Model for Image Animation (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.