Research map / Video / Video generation
AnyMoLe: Any Character Motion In-betweening Leveraging Video Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Performs 3D character motion in-betweening for arbitrary rigged characters without external motion data by fine-tuning a video interpolation diffusion model on rendered context frames and fitting joints to the generated video.
Problem
Learning-based motion in-betweening requires large character-specific motion datasets, so it cannot be applied to arbitrary characters, including non-humanoids.
Approach
Uses a pretrained image-to-video interpolation diffusion model (DynamiCrafter) on rendered keyframes, with context-frame inpainting, two-stage coarse-to-fine generation, per-scene fine-tuning (ICAdapt), and motion-video mimicking via a scene-specific 2D/3D joint estimator built on DINOv2 and FiT3D features.
Key results
- On humanoid characters, AnyMoLe reached HL2Q 0.0015 and L2P 0.0063, versus 0.0971 and 0.3733 for SinMDM.
- On non-humanoid characters, AnyMoLe reached LPIPS 0.0433 and SSIM 0.9531 versus SinMDM's 0.0816 and 0.9362.
- In a 21-person user study, AnyMoLe was preferred in 60-91% of selections across similarity, faithfulness, and naturalness criteria.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AnyMoLe instead of the strongest reported baseline | motion in-betweening · Humanoid | HL2Q: 0.0028 -> 0.0015 | Helped |
| Use AnyMoLe instead of the strongest reported baseline | motion in-betweening · Humanoid | L2Q: 0.0106 -> 0.0011 | Helped |
| Use AnyMoLe instead of the strongest reported baseline | motion in-betweening · Non-humanoid | L2P: 0.1382 -> 0.0299 | Helped |
| Remove ICAdapt (ablation) | AnyMoLe · Humanoid and non-humanoid | HL2Q: 0.00169 -> 0.00210 | Hurt |
| Remove fine-stage video generation (ablation) | AnyMoLe · Humanoid and non-humanoid | HL2Q: 0.00169 -> 0.00200 | Hurt |
| Replace the scene-specific joint estimator with XPose (ablation) | AnyMoLe · Humanoid and non-humanoid | HL2Q: 0.00169 -> 0.00174 | Hurt |
| Remove keyframe weighting and back-view exclusion from joint-estimator training (ablation) | AnyMoLe · Humanoid and non-humanoid | HL2Q: 0.00169 -> 0.00799 | Hurt |
Limitations
- Full pipeline takes five to six hours per character/motion
- Fast or complex motions produce blurry generated frames that cause left/right joint ambiguity
- Scene-specific joint estimator must be retrained for each character
Experimental setup
Datasets: Mixamo Truebones Zoo
Benchmarks: Mixamo humanoid characters and Truebones Zoo non-humanoid characters (20 motions)
Metrics: HL2Q L2Q L2P NPSS LPIPS CLIP similarity SSIM
Baselines: SinMDM SinMDM* TST ERD-QV XPose
Builds on: DynamiCrafter DINOv2 FiT3D SinMDM
Closest papers by meaning
- Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (CVPR 2024 · similarity 0.90)
- Animate-X: Universal Character Image Animation with Enhanced Motion Representation (ICLR 2025 · similarity 0.87)
- Single Motion Diffusion (ICLR 2024 · similarity 0.87)
- MIMO: Controllable Character Video Synthesis with Spatial Decomposed Modeling (CVPR 2025 · similarity 0.86)
- Arbitrary Motion Style Transfer with Multi-condition Motion Latent Diffusion Model (CVPR 2024 · similarity 0.85)
- TapMo: Shape-aware Motion Generation of Skeleton-free Characters (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.