Research map / 3D vision / Human pose estimation
3D Human Pose Perception from Egocentric Stereo Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a transformer-based framework for egocentric stereo 3D human pose estimation using SfM-reconstructed scene depth with padding masks and video-dependent joint query augmentation, and releases two new datasets (UnrealEgo2, UnrealEgo-RW).
Problem
Estimating accurate 3D full-body poses from eyeglasses-mounted egocentric stereo fisheye cameras is hard due to self-occlusion and depth ambiguity, and existing stereo datasets are small, synthetic-only, or lack diversity.
Approach
Combine SfM-reconstructed background scene depth (with depth padding masks for missing values) and 2D joint heatmaps in a DETR-style transformer decoder whose joint queries are augmented with pooled stereo-video features, trained with current and past frame pose supervision.
Key results
- On UnrealEgo2, device-relative MPJPE drops to 30.53 mm vs 72.80 mm for Akada et al. (>=40% improvement).
- On UnrealEgo-RW, MPJPE improves to 104.14 mm from 115.36 mm for the baseline and 116.57 for prior methods (>10%).
- Pre-training on UnrealEgo2 and fine-tuning on UnrealEgo-RW reduces MPJPE to 72.89 mm for the proposed method (from 75.34 mm with UnrealEgo pre-training).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add depth information to the CNN-based 3D module (ablation) | Baseline · UnrealEgo-RW | MPJPE: 115.36 -> 120.39 | Hurt |
| Add video-dependent query adaptation (ablation) | Ours · UnrealEgo-RW | MPJPE: 108.33 -> 104.14 | Helped |
| Add depth information (ablation) | Ours · UnrealEgo-RW | MPJPE: 112.56 -> 104.14 | Helped |
| Add the depth padding mask (ablation) | Ours · UnrealEgo-RW | MPJPE: 108.70 -> 104.14 | Helped |
| Supervise past poses as well as the latest pose (ablation) | Ours · UnrealEgo-RW | MPJPE: 105.67 -> 104.14 | Helped |
| Use T sets of joint queries instead of one set (ablation) | Ours · UnrealEgo-RW | MPJPE: 105.58 -> 104.14 | Helped |
| Add depth information (ablation) | Ours · UnrealEgo-RW, lower body | MPJPE: 144.31 -> 130.97 | Helped |
| Add depth padding masks (ablation) | Ours · UnrealEgo-RW, lower body | MPJPE: 140.10 -> 130.97 | Helped |
| Increase the input sequence length from T=1 to T=5 (ablation) | Ours · UnrealEgo-RW | MPJPE: 108.63 -> 104.14 | Helped |
| Increase the input sequence length from T=3 to T=5 (ablation) | Ours · UnrealEgo-RW | MPJPE: 108.23 -> 104.14 | Helped |
| Increase the input sequence length from T=5 to T=7 (ablation) | Ours · UnrealEgo-RW | MPJPE: 104.14 -> 104.01 | No clear effect |
Surprising findings
- Naively concatenating depth maps to the CNN-based baseline does not improve accuracy (120.39 mm vs 115.36 mm MPJPE).
- A single-frame variant (T=1) of the proposed transformer already outperforms the best existing temporal method, suggesting autoencoder-based 3D modules are suboptimal.
Limitations
- 3D scene and camera poses cannot always be reconstructed under severe self-occlusion, leaving missing depth values.
- Real-world dataset is recorded in a multi-view motion capture studio with 16 identities and 1.5 hours of motion.
Experimental setup
Datasets: UnrealEgo UnrealEgo2 UnrealEgo-RW
Benchmarks: UnrealEgo UnrealEgo2 UnrealEgo-RW
Metrics: MPJPE PA-MPJPE 3D PCK AUC MPE
Baselines: Akada et al. (UnrealEgo) Zhao et al. (EgoGlass) Kang et al. Baseline (temporal UnrealEgo variant)
Builds on: UnrealEgo DETR Direct multi-view multi-person 3D human pose estimation (joint query adaptation) Segment Anything
Closest papers by meaning
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.89)
- FreeMan: Towards Benchmarking 3D Human Pose Estimation under Real-World Conditions (CVPR 2024 · similarity 0.88)
- Attention-Propagation Network for Egocentric Heatmap to 3D Pose Lifting (CVPR 2024 · similarity 0.88)
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.88)
- FRAME: Floor-aligned Representation for Avatar Motion from Egocentric Video (CVPR 2025 · similarity 0.87)
- FusionFormer: A Concise Unified Feature Fusion Transformer for 3D Pose Estimation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.