Research map / Image generation / Fast diffusion sampling
REWIND: Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes REWIND, a causal one-step diffusion model for real-time egocentric whole-body (body and hands) motion estimation from stereo images, with cascaded body-hand denoising, a causal relative-temporal Transformer, score-distillation, and exemplar-based identity conditioning.
Problem
Real-time, causal, high-fidelity whole-body motion estimation from egocentric stereo images; existing EgoWholeMocap is non-real-time and acausal due to iterative diffusion refinement.
Approach
Cascade separate body and hand diffusion models (hand conditioned on upper-body motion), use a causal windowed attention Transformer with RoPE relative timesteps for length generalization, distill to a single denoising step, and condition on a small set of identity pose exemplars via AdaIN.
Key results
- On ColossusEgo, REWIND reduces body MPJPE from 62.49 mm (EgoWholeMocap) to 53.83 mm, and with exemplar identity conditioning reaches 48.45 mm.
- On UnrealEgo, REWIND achieves body MPJPE 37.23 mm vs 49.10 mm for EgoWholeMocap.
- Single-step distilled model runs at 32 ms inference on an A100 (over 30 FPS), versus 2576 ms for EgoWholeMocap.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use REWIND instead of EgoWholeMocap | egocentric whole-body motion estimation · ColossusEgo | MPJPE Body: 62.49 -> 53.83 | Helped |
| Use REWIND instead of EgoWholeMocap | egocentric whole-body motion estimation · UnrealEgo | MPJPE Body: 49.10 -> 37.23 | Helped |
| Add cascaded body-hand estimation rather than estimating whole-body keypoints jointly (ablation) | REWIND · ColossusEgo | MPJPE Body: 55.09 -> 53.83 | Helped |
| Use diffusion rather than regression (ablation) | REWIND · ColossusEgo | MPJPE Body: 57.34 -> 53.83 | Helped |
| Use non-autoregressive causal relative-temporal modeling instead of autoregressive modeling (ablation) | REWIND · ColossusEgo | MPJPE Body: 58.14 -> 53.83 | Helped |
| Add diffusion distillation for one-step sampling (ablation) | REWIND · ColossusEgo | MPJPE Body: 56.12 -> 53.83 | Helped |
| Use multi-step teacher sampling instead of one-step distilled sampling (ablation) | REWIND · ColossusEgo | MPJPE Body: 53.83 -> 46.18 | Helped |
| Add exemplar-based identity conditioning using 10 estimated example poses (ablation) | REWIND · ColossusEgo | MPJPE Body: 53.83 -> 48.45 | Helped |
| Add height conditioning (ablation) | REWIND · ColossusEgo | MPJPE Body: 53.83 -> 51.98 | Helped |
| Add shape-parameter conditioning (ablation) | REWIND · ColossusEgo | MPJPE Body: 53.83 -> 51.40 | Helped |
| Add bone-length conditioning (ablation) | REWIND · ColossusEgo | MPJPE Body: 53.83 -> 49.74 | Helped |
| Use 10 ground-truth exemplar poses instead of estimated exemplar poses (ablation) | REWIND with exemplar identity conditioning · ColossusEgo | MPJPE Body: 48.45 -> 38.99 | Helped |
Surprising findings
- Using ground-truth example poses for identity conditioning (Exemplar†) drops body MPJPE to 38.99 mm, indicating large headroom from better identity priors.
- Diffusion distillation with score distillation sampling alone, without an adversarial loss, suffices for SotA results.
Limitations
- Small portion of reconstructed motions show self-penetrations.
- Identity conditioning requires a registration stage to acquire example poses from monocular photos.
Experimental setup
Datasets: ColossusEgo UnrealEgo1 UnrealEgo2
Benchmarks: ColossusEgo UnrealEgo
Metrics: MPJPE PA-MPJPE Bone Error Foot Skate
Baselines: EgoWholeMocap EgoPoseFormer
Builds on: EgoWholeMocap FlowMDM DDPM DDIM Score Distillation Sampling
Closest papers by meaning
- Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data (NeurIPS 2024 · similarity 0.89)
- Egocentric Whole-Body Motion Capture with FisheyeViT and Diffusion-Based Motion Refinement (CVPR 2024 · similarity 0.87)
- Whole-Body Conditioned Egocentric Video Prediction (NeurIPS 2025 · similarity 0.86)
- DiffusionPoser: Real-time Human Motion Reconstruction From Arbitrary Sparse Sensors Using Autoregressive Diffusion (CVPR 2024 · similarity 0.86)
- CoDA: Coordinated Diffusion Noise Optimization for Whole-Body Manipulation of Articulated Objects (NeurIPS 2025 · similarity 0.86)
- Estimating Body and Hand Motion in an Ego-sensed World (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.