Research map / Video / Video segmentation and tracking
FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the FRAME dataset (7.4 h, 1.6M frames) from a VR-based stereo body-facing rig with on-device 6D tracking, and a floor-aligned, stereo temporal fusion model for real-time egocentric body pose estimation.
Problem
Egocentric full-body motion capture from head-mounted body-facing stereo fisheye cameras suffers from heavy occlusion, scarce real-world training data, and lower-limb artifacts such as foot penetration and sliding.
Approach
Use the VR headset's on-device 6D pose to rototranslate per-camera 3D joint predictions into a floor- and gravity-aligned frame, fuse stereo views and past predictions with a transformer (STF), and train the temporal module on k-fold cross-training cached backbone predictions to mimic realistic errors.
Key results
- FRAME achieves 47.53 mm MPJPE vs 69.18 mm for EgoPoseFormer (about 28% lower) on the FRAME test set.
- The model reaches 100% Non-Penetration Percentage and 2.68 ms inference (about 300 FPS on an RTX 3090).
- Ablation: moving predictions to the floor-aligned frame reduces MPJPE from 71.31 to 59.53 mm; cross-training caching and STF further reduce it to 47.53 mm.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FRAME instead of EgoPoseFormer | egocentric pose estimation · SELF test set | MPJPE (mm): 69.18 -> 47.53 | Helped |
| use FRAME instead of EgoPoseFormer | egocentric pose estimation · SELF test set | 3D-PCK (%): 78.98 -> 92.56 | Helped |
| use FRAME instead of EgoPoseFormer | egocentric pose estimation · SELF test set | NPP (%): 49.45 -> 100.0 | Helped |
| use FRAME instead of EgoPoseFormer | egocentric pose estimation · SELF test set | Inference (ms): 14.36 -> 2.68 | Helped |
| remove the right-camera view and stereo information (ablation) | FRAME · SELF test set | MPJPE (mm): 47.53 -> 87.94 | Hurt |
| merge aligned stereo predictions by averaging (ablation) | FRAME · SELF test set | MPJPE (mm): 87.94 -> 84.31 | Helped |
| merge left and right predictions with an MLP without explicit alignment (ablation) | FRAME · SELF test set | MPJPE (mm): 84.31 -> 82.19 | Helped |
| explicitly align stereo predictions before MLP fusion (ablation) | FRAME · SELF test set | MPJPE (mm): 82.19 -> 81.69 | Helped |
| add a shared head for joint processing of image features (ablation) | FRAME · SELF test set | MPJPE (mm): 81.69 -> 74.13 | Helped |
| use learnable soft-argmax (ablation) | FRAME · SELF test set | MPJPE (mm): 74.13 -> 71.31 | Helped |
| rototranslate predictions into floor-aligned frame F (ablation) | FRAME · SELF test set | MPJPE (mm): 71.31 -> 59.53 | Helped |
| add Cross Training Caching (ablation) | FRAME · SELF test set | MPJPE (mm): 59.53 -> 54.08 | Helped |
Surprising findings
- Re-projecting the existing monocular EgoWholeBody model into the floor-aligned frame reduces its MPJPE from 85.63 to 76.43 mm, showing the frame choice is useful beyond this method.
Limitations
- Does not exploit other VR modalities such as environment meshes, forward-facing cameras, eye gaze, or controller hand poses.
- Struggles with significant occlusions and self-contact.
- Physics-based constraints are suggested as future work rather than integrated.
Experimental setup
Datasets: FRAME
Benchmarks: FRAME test set
Metrics: MPJPE PA-MPJPE 3D-PCK Jitter NPP MPE FS Inference time
Baselines: EgoGlass UnrealEgo EgoPoseFormer EgoWholeBody
Builds on: UnrealEgo EgoWholeBody EgoPoseFormer
Closest papers by meaning
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.91)
- EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs (AAAI 2025 · similarity 0.89)
- Instance Tracking in 3D Scenes from Egocentric Videos (CVPR 2024 · similarity 0.89)
- Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input (CVPR 2025 · similarity 0.88)
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.88)
- EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Videos Generation (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.