Research map / 3D vision / Human pose estimation
ExtPose: Robust and Coherent Pose Estimation by Extending ViTs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends a pre-trained image ViT pose estimator (HMR2.0/HaMeR) with dual-stream image + skeleton-image full attention and 3D spatiotemporal attention across frames, without extra parameters, to improve pixel alignment and temporal coherence for human and hand pose estimation.
Problem
3D human and hand pose estimation with ViTs is limited to single images, lacks temporal information, and often produces predictions misaligned with the 2D image evidence.
Approach
Represents 2D poses as rendered skeleton images processed by the same shared ViT, uses cross-modal self-attention with a mask between image and skeleton streams, and extends 2D attention to 3D video attention across frames, reusing pretrained weights with no added modules.
Key results
- On 3DPW (image, T=1), PA-MPJPE improves from 44.4mm (HMR2.0) to 35.5mm; video with T=16 reaches 34.0mm PA-MPJPE, 23% better than prior ViT methods per the abstract.
- On FreiHAND, PA-MPJPE drops to 4.9mm from 6.0mm for HaMeR (about -14% vs. prior SOTA in Table 2 and -18% per abstract).
- On HO3D(v2) image setting, PA-MPJPE is 7.0 vs 7.7 for HaMeR (-9.1%); on HInt NEW DAYS @0.05 PCK, EXTPOSE reaches 59.6 vs 48.0 for HaMeR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add 2D pose evidence to HMR2.0 | EXTPOSE · 3DPW image | PA-MPJPE: 44.4 -> 35.5 | Helped |
| Extend attention across video frames | EXTPOSE · 3DPW video | PA-MPJPE: 35.9 -> 34.0 | Helped |
| Add 2D pose evidence with cross-modal attention | HaMeR · FreiHAND | PA-MPJPE: 6.0 -> 4.9 | Helped |
| Add 2D pose evidence with cross-modal attention | HaMeR · HO3D image | PA-MPJPE: 7.7 -> 7.0 | Helped |
| Use only 1D 2D-joint coordinates instead of image and skeleton-image inputs (ablation) | EXTPOSE · FreiHAND | PA-MPJPE: 4.9 -> 6.5 | Hurt |
| Use only 2D pose heatmaps instead of image and skeleton-image inputs (ablation) | EXTPOSE · FreiHAND | PA-MPJPE: 4.9 -> 6.3 | Hurt |
| Use only a 2D skeleton image instead of image and skeleton-image inputs (ablation) | EXTPOSE · FreiHAND | PA-MPJPE: 4.9 -> 6.2 | Hurt |
| Remove the 2D pose input (ablation) | EXTPOSE · FreiHAND | PA-MPJPE: 4.9 -> 6.0 | Hurt |
| Replace cross-modal attention with late fusion (ablation) | EXTPOSE · HInt New Days | PCK @0.05: 59.6 -> 50.5 | Hurt |
| Replace cross-modal attention with channel concatenation (ablation) | EXTPOSE · HInt New Days | PCK @0.05: 59.6 -> 56.3 | Hurt |
| Replace cross-modal attention with ControlNet fusion (ablation) | EXTPOSE · HInt New Days | PCK @0.05: 59.6 -> 55.6 | Hurt |
| Initialize from ViTPose instead of the HPE baseline (ablation) | EXTPOSE · HInt New Days | PCK @0.05: 59.6 -> 49.9 | Hurt |
Surprising findings
- Zero-shot 3D attention applied to a pretrained image model already improves temporal coherence without any training.
- Video-based ViT-style methods had previously underperformed image-based methods on frame-wise accuracy; this approach reverses that trend on 3DPW.
Limitations
- Reconstruction limited to body or hand; whole-body extension left to future work.
- Relies on off-the-shelf 2D detector (ViTPose) quality; 2D pose estimates can be inaccurate and degrade results.
- Video models still lag in some settings; full-attention over frames and modalities increases compute cost with sequence length.
Experimental setup
Datasets: 3DPW Human3.6M MPI-INF-3DHP COCO FreiHAND HO3D MTC RHD InterHand2.6M H2O3D DexYCB COCO WholeBody
Benchmarks: 3DPW FreiHAND HO3D HInt
Metrics: MPVPE MPJPE PA-MPJPE PA-MPVPE F@5 F@15 AUCJ AUCV PCK@0.05 PCK@0.1
Baselines: HMR2.0 HaMeR WHAM TokenHMR ReFit CLIFF Deformer VIBE TCMR ViTPose MeshGraphormer MobRecon
Builds on: HMR2.0 HaMeR ViTPose ViT SMPL
Closest papers by meaning
- SVTformer: Spatial-View-Temporal Transformer for Multi-View 3D Human Pose Estimation (AAAI 2025 · similarity 0.88)
- Just Add ?! Pose Induced Video Transformers for Understanding Activities of Daily Living (CVPR 2024 · similarity 0.86)
- Lifting by Image – Leveraging Image Cues for Accurate 3D Human Pose Estimation (AAAI 2024 · similarity 0.86)
- HiPART: Hierarchical Pose AutoRegressive Transformer for Occluded 3D Human Pose Estimation (CVPR 2025 · similarity 0.85)
- SHaRPose: Sparse High-Resolution Representation for Human Pose Estimation (AAAI 2024 · similarity 0.85)
- Semantics-aware Test-time Adaptation for 3D Human Pose Estimation (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.