Research map / 3D vision / 3D generation
From Sparse Signal to Smooth Motion: Real-Time Motion Generation with Rolling Prediction Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Rolling Prediction Model (RPM), an online autoregressive full-body motion generator from sparse XR tracking that uses a Prediction Consistency Anchor Function and free-running training, and releases the GORP real VR dataset.
Problem
Generating smooth, real-time full-body human motion from temporally and spatially sparse XR tracking inputs (hand tracking with frequent signal loss, or motion controllers), where existing methods snap abruptly when tracking reappears and models trained on synthetic inputs fail on real data.
Approach
Predict a window of future poses at each step and refine the previous prediction through a PCAF reparameterization (previous prediction plus bounded tanh correction weighted by a per-frame uncertainty schedule), so tracking-induced changes are injected gradually; combined with free-running training to handle its own errors.
Key results
- On A-P1 hand-tracking setup, RPM-Smooth reduces AUJ S-T to 69.02 vs 2133.20 for AvatarPoser (roughly 30x lower), with jitter 4.21-4.35 vs 26.10 for AvatarPoser.
- PCAF halves jitter (from 8.32 to 4.35) and reduces AUJ T-S and S-T by seven and elevenfold in ablation on A-P1 HT.
- RPM runs in real time at 206.5 FPS, trained on a single A100 in 12 hours.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add free-running training (ablation) | RPM · A-P1 HT | MPJRE: 6.69 -> 3.79 | Helped |
| Add free-running training (ablation) | RPM · A-P1 HT | Jitter: 5.84 -> 8.32 | Hurt |
| Add PCAF (ablation) | RPM with free-running · A-P1 HT | Jitter: 8.32 -> 4.35 | Helped |
| Add PCAF (ablation) | RPM with free-running · A-P1 HT | AUJ T-S: 428.32 -> 60.51 | Helped |
| Add PCAF (ablation) | RPM with free-running · A-P1 HT | AUJ S-T: 799.03 -> 69.02 | Helped |
| Use a longer prediction window | RPM · A-P1 | Jitter | Helped |
| Use a longer prediction window | RPM · A-P1 | hands PE | Hurt |
| Train on real instead of simulated motion-controller inputs | RPM and evaluated methods · GORP real MC | MPJPE: 7.36 -> 6.83 | Mixed |
| Train on real instead of simulated hand-tracking inputs | RPM and evaluated methods · GORP real HT | MPJPE: 7.95 -> 7.57 | Hurt |
Surprising findings
- Models trained on synthetic MC/HT inputs see MPJPE increase about 20% when evaluated on real MC input, and jitter rises up to 4x for AvatarJLM on real HT input.
Limitations
- RPM is deterministic, so generated motion can lose expressivity during long tracking losses.
- Synthetic-trained models underperform on real GORP inputs, showing a sim-to-real gap.
Experimental setup
Datasets: AMASS GORP
Benchmarks: AMASS A-P1 AMASS A-P2 GORP
Metrics: MPJRE MPJPE MPJVE Jitter Peak Jerk Area Under the Jerk
Baselines: AvatarPoser AGRoL AvatarJLM SAGE EgoPoser HMD-Poser
Builds on: AvatarPoser AGRoL HMD-Poser SMPL-X
Closest papers by meaning
- A Unified Diffusion Framework for Scene-aware Human Motion Estimation from Sparse Signals (CVPR 2024 · similarity 0.87)
- EnvPoser: Environment-aware Realistic Human Motion Estimation from Sparse Observations with Uncertainty Modeling (CVPR 2025 · similarity 0.86)
- DiffGrasp: Whole-Body Grasping Synthesis Guided by Object Motion Using a Diffusion Model (AAAI 2025 · similarity 0.85)
- Hand-Centric Motion Refinement for 3D Hand-Object Interaction via Hierarchical Spatial-Temporal Modeling (AAAI 2024 · similarity 0.85)
- HandDiffuse: Generative Controllers for Two-Hand Interactions via Diffusion Models (AAAI 2025 · similarity 0.85)
- CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objects (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.