Research map / 3D vision / Human pose estimation
Whole-Body Conditioned Egocentric Video Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains an autoregressive conditional diffusion transformer (PEVA) that predicts egocentric video from whole-body 3D pose action sequences on Nymeria, with a hierarchical evaluation from atomic actions to planning.
Problem
Predicting first-person video consequences of whole-body human motion, which is needed for embodied simulation and planning, is underexplored; prior world models use low-dimensional controls and ignore body dynamics.
Approach
Condition an autoregressive diffusion transformer (CDiT-style) on structured 48-D relative joint-rotation and root-translation actions following the Xsens kinematic hierarchy, with random timeskips and sequence-prefix training with causal attention masks.
Key results
- PEVA-XXL reaches LPIPS 0.298, DreamSim 0.186 and FID 61.100 at 2-second single-step prediction, improving with scale from PEVA-S (LPIPS 0.370).
- Increasing context from 3 to 15 frames lowers DreamSim from 0.199 to 0.193 at 2-second prediction.
- DreamSim degrades gradually from 0.178 at 1s to 0.390 at 16s in long-horizon rollouts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PEVA-XXL instead of DF* | PEVA · Atomic actions (all 11 action types) | score: 0.393^{0.011} -> 0.325^{0.006}; 0.314^{0.006} -> 0.269^{0.005}; 0.279^{0.005} -> 0.234^{0.004}; 0.292^{0.009} -> 0.236^{ | Helped |
| Increase context length from 3 to 15 frames (ablation) | PEVA · Nymeria validation, 2 seconds ahead | LPIPS: 0.304^{0.002} -> 0.303^{0.001} | Helped |
| Increase context length from 7 to 15 frames (ablation) | PEVA · Nymeria validation, 2 seconds ahead | LPIPS: 0.304^{0.001} -> 0.303^{0.001} | Helped |
| Use Action Concatenation instead of Action Embedding (d=512) (ablation) | PEVA · Nymeria validation, 2 seconds ahead | LPIPS: 0.317^{0.003} -> 0.303^{0.001} | Helped |
| Increase model size from PEVA-S to PEVA-XXL (ablation) | PEVA · Nymeria validation, 2 seconds ahead | LPIPS: 0.370^{0.002} -> 0.298^{0.002} | Helped |
| Increase model size from PEVA-B to PEVA-XXL (ablation) | PEVA · Nymeria validation, 2 seconds ahead | LPIPS: 0.337^{0.001} -> 0.298^{0.002} | Helped |
| Increase model size from PEVA-L to PEVA-XXL (ablation) | PEVA · Nymeria validation, 2 seconds ahead | LPIPS: 0.308^{0.002} -> 0.298^{0.002} | Helped |
| Increase model size from PEVA-XL to PEVA-XXL (ablation) | PEVA · Nymeria validation, 2 seconds ahead | LPIPS: 0.303^{0.001} -> 0.298^{0.002} | Helped |
Surprising findings
- Simple concatenation of the action vector outperforms an MLP-based action embedding (LPIPS 0.303 vs 0.317).
- Planning results are described as preliminary, limited to controlling a single arm via CEM, with some rollouts failing to reach the goal.
Limitations
- Planning evaluation is preliminary and only controls one arm via CEM, with no long-horizon or closed-loop control.
- Lacks conditioning on task intent or semantic goals; image similarity (LPIPS) is used as proxy objective.
- Some planning rollouts fail to match goals because only one arm is predicted and no whole-body coordination is modeled.
Experimental setup
Datasets: Nymeria
Benchmarks: Nymeria validation split atomic action generation set (100 examples per action type)
Metrics: LPIPS DreamSim FID PSNR
Baselines: CDiT Diffusion Forcing (DF*)
Builds on: CDiT Diffusion Forcing DDPM Navigation World Models
Closest papers by meaning
- Estimating Ego-Body Pose from Doubly Sparse Egocentric Video Data (NeurIPS 2024 · similarity 0.88)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.87)
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.87)
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation (NeurIPS 2025 · similarity 0.87)
- REWIND: Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioning (CVPR 2025 · similarity 0.86)
- EgoExo-Gen: Ego-centric Video Prediction by Watching Exo-centric Videos (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.