Research map / Image generation / Editing and style transfer
Continuous Piecewise-Affine Based Motion Model for Image Animation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CPAB-based motion modeling for unsupervised image animation, plus a SAM-guided keypoint semantic loss and a DINO-based structure alignment loss, built on the TPSMM framework.
Problem
Unsupervised image animation methods using affine or thin-plate-spline transformations produce poor results when the motion gap between source and driving frames is large, and their detected keypoints are semantically inconsistent.
Approach
Model source-to-driving motion with continuous piecewise-affine based (CPAB) diffeomorphic transformations fitted to keypoints by gradient descent, add a global CPAB transformation over all keypoints, supervise keypoint semantics with SAM embeddings, and align DINO ViT self-similarity of keys between generated and driving frames.
Key results
- On TaiChiHD video reconstruction, the method reaches L1 0.041 and AED 0.117, versus 0.045 and 0.151 for TPSMM.
- On TED-talks, L1 improves to 0.022 from TPSMM's 0.027, and AED to 0.085 from 0.124.
- User study with 35 volunteers preferred the method on all four datasets, with 56.57%, 50.86%, 55.43% and 65.71% of votes; AKD and MKR on TaiChiHD and VoxCeleb are slightly worse than TPSMM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace TPS transformations with CPAB transformations (ablation) | TPSMM · TaiChiHD | AED: 0.151 -> 0.123 | Helped |
| replace TPS transformations with CPAB transformations (ablation) | TPSMM · TaiChiHD | AKD: 4.57 -> 5.10 | Hurt |
| add a global CPAB transformation (ablation) | +CPAB · TaiChiHD | AKD: 5.10 -> 4.96 | Helped |
| add a global CPAB transformation (ablation) | +CPAB · TaiChiHD | MKR: 0.024 -> 0.023 | Helped |
| add the SAM-guided keypoint semantic loss (ablation) | +CPAB,Global · TaiChiHD | L1: 0.044 -> 0.043 | Helped |
| add the SAM-guided keypoint semantic loss (ablation) | +CPAB,Global · TaiChiHD | MKR: 0.023 -> 0.024 | Hurt |
| add the DINO structure alignment loss (ablation) | +CPAB,Global · TaiChiHD | L1: 0.044 -> 0.042 | Helped |
| add the DINO structure alignment loss (ablation) | +CPAB,Global · TaiChiHD | AKD: 4.96 -> 4.81 | Helped |
| add the DINO structure alignment loss (ablation) | +CPAB,Global · TaiChiHD | MKR: 0.023 -> 0.023 | No clear effect |
| use the full CPAB model rather than TPSMM (ablation) | TaiChiHD reconstruction · TaiChiHD | AED: 0.151 -> 0.117 | Helped |
Surprising findings
- CPAB transformation gives slightly worse AKD and MKR than TPS on TaiChiHD and VoxCeleb, attributed to TPS's stronger keypoint alignment at the cost of image distortion.
- Adding the keypoint semantic loss slightly increased MKR, which the authors describe as unexpected.
Limitations
- AKD and MKR are slightly worse than TPSMM on TaiChiHD and VoxCeleb.
- Training takes about seven to fifteen days on a single RTX 3090 per dataset.
- Keypoint-based CPAB fitting requires 200 gradient descent iterations per keypoint set.
Experimental setup
Datasets: TaiChiHD TED-talks VoxCeleb MGif
Benchmarks: TaiChiHD TED-talks VoxCeleb MGif
Metrics: L1 AKD MKR AED
Baselines: X2Face Monkey-Net FOMM MRAA TPSMM
Builds on: TPSMM first-order motion model Segment Anything DINO ViT CPAB transformations
Closest papers by meaning
- Motion Representations for Articulated Animation (CVPR 2021 · similarity 0.87)
- MotionPro: A Precise Motion Controller for Image-to-Video Generation (CVPR 2025 · similarity 0.85)
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance (NeurIPS 2025 · similarity 0.85)
- Animate-X: Universal Character Image Animation with Enhanced Motion Representation (ICLR 2025 · similarity 0.84)
- Boximator: Generating Rich and Controllable Motions for Video Synthesis (ICML 2024 · similarity 0.84)
- Enhanced Fine-Grained Motion Diffusion for Text-Driven Human Motion Synthesis (AAAI 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.