Research map / 3D vision / Human pose estimation
Attention-Propagation Network for Egocentric Heatmap to 3D Pose Lifting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EgoTAP, a heatmap-to-3D lifting method for stereo egocentric pose estimation that combines a Grid ViT heatmap encoder with a skeleton-propagating LSTM-style network, reducing MPJPE by 23.9% on UnrealEgo.
Problem
Accurate 3D full-body pose estimation from stereo head-mounted egocentric cameras is hard due to self-occlusion and out-of-view limbs, and existing heatmap-to-3D lifting loses information in feature embedding and ignores joint certainty and skeletal structure.
Approach
Encode each joint heatmap as a patch in a grid image with a ViT so that per-joint embeddings preserve heatmap correspondence, then propagate features down the skeleton tree with a Propagation Unit that uses a joint-feature forget gate to weight parent and limb information only for obscure joints.
Key results
- On UnrealEgo, EgoTAP reaches 41.06 MPJPE vs 53.99 for Ego3DPose (23.9% reduction) and 35.39 PA-MPJPE.
- On EgoCap, EgoTAP reaches 55.38 MPJPE, a 22.6% improvement over EgoGlass's reported 67.90 and 9.4% lower PA-MPJPE than Ego3DPose.
- Replacing the CNN encoder with the Grid ViT reduces heatmap reconstruction MSE from 4.84e-4 to 1.68e-4 per pixel and MPJPE from 63.53 to 49.03 on UnrealEgo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EgoTAP instead of Ego3DPose | stereo egocentric pose estimation · UnrealEgo test | MPJPE: 53.99 -> 41.06 | Helped |
| use EgoTAP instead of Ego3DPose | stereo egocentric pose estimation · UnrealEgo test | PA-MPJPE: 43.02 -> 35.39 | Helped |
| use EgoTAP instead of Ego3DPose | stereo egocentric pose estimation · EgoCap test | MPJPE: 69.45 -> 55.38 | Helped |
| use EgoTAP instead of Ego3DPose | stereo egocentric pose estimation · EgoCap test | PA-MPJPE: 49.98 -> 45.24 | Helped |
| replace Grid ViT with channel-wise ViT encoder (ablation) | EgoTAP · UnrealEgo test | MPJPE: 49.03 -> 61.62 | Hurt |
| replace Grid ViT with channel-wise ViT encoder (ablation) | EgoTAP · EgoCap test | MPJPE: 63.97 -> 83.39 | Hurt |
| add relational features without propagation (ablation) | Grid ViT · UnrealEgo test | MPJPE: 49.03 -> 48.12 | Helped |
| add relational features without propagation (ablation) | Grid ViT · EgoCap test | MPJPE: 63.97 -> 63.09 | Helped |
| add LSTM propagation using joint features only (ablation) | Grid ViT · UnrealEgo test | MPJPE: 49.03 -> 49.43 | Hurt |
| add LSTM propagation using joint features only (ablation) | Grid ViT · EgoCap test | MPJPE: 63.97 -> 60.16 | Helped |
| replace PU with LSTM alternating relational and joint features (ablation) | EgoTAP · UnrealEgo test | MPJPE: 49.03 -> 44.97 | Helped |
| replace PU with LSTM using concatenated relational and joint features (ablation) | EgoTAP · EgoCap test | MPJPE: 63.97 -> 58.35 | Helped |
Surprising findings
- Simply applying a standard channel-concatenated ViT degrades performance on EgoCap (MPJPE 83.39 vs 70.77 for CNN), showing correspondence between patches and heatmaps matters.
- Relational (limb) features alone without propagation give marginal or negative gains.
Limitations
- Performance on EgoCap is weaker because estimated limb heatmaps are less accurate there, attributed to limited data and subjects.
- Gains are smaller for highly occluded activities such as crawling.
- Propagation cannot improve child joints when limbs are occluded, yielding near-zero effect in some cases.
Experimental setup
Datasets: UnrealEgo EgoCap
Benchmarks: UnrealEgo EgoCap
Metrics: MPJPE PA-MPJPE
Baselines: EgoGlass UnrealEgo Ego3DPose CNN encoder Channel ViT
Builds on: Vision Transformer LSTM UnrealEgo Ego3DPose xR-EgoPose
Closest papers by meaning
- 3D Human Pose Perception from Egocentric Stereo Videos (CVPR 2024 · similarity 0.88)
- EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity (NeurIPS 2024 · similarity 0.86)
- PandaPose: 3D Human Pose Lifting from a Single Image via Propagating 2D Pose Prior to 3D Anchor Space (NeurIPS 2025 · similarity 0.86)
- Neural Localizer Fields for Continuous 3D Human Pose and Shape Estimation (NeurIPS 2024 · similarity 0.86)
- Lifting by Image – Leveraging Image Cues for Accurate 3D Human Pose Estimation (AAAI 2024 · similarity 0.86)
- HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.