Research map / 3D vision / 3D generation
Pose-Guided Temporal Enhancement for Robust Low-Resolution Hand Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a temporal enhancement framework that fuses joint features from past frames into current visual features via triplane encoding, enabling accurate 3D hand reconstruction from 64x64 (and 32x32) video frames.
Problem
3D hand reconstruction from low-resolution hand crops (e.g., 64x64 at longer interaction distances) performs poorly due to lost appearance detail, joint self-similarity and depth ambiguity.
Approach
Temporal joint features from previous frames are projected onto three orthogonal planes (triplane) to form dense 3D-aware maps, which are compressed via spatial-aware fusion into 1D global cues and fused into the current frame's 2D visual features before mesh regression.
Key results
- On DexYCB at 64x64 input, the method reaches 5.72 PA-MPJPE, beating the high-resolution (256 px) monocular and sequence baselines in Table 1 except Deformer (512 px, 5.22).
- On HanCo, the method achieves 5.77 J-PA, outperforming the monocular HaMeR (5.99) at 256 px per Table 3.
- On H2O, the method reaches 18.15 MPJPE and 6.11 PA-MPJPE, versus 16.19 and 6.13 for HandOccNet (Table 2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add temporal enhancement to monocular input (ablation) | Pose-guided temporal framework · HanCo | J-PA: 6.19 -> 5.77 | Helped |
| use 6 finger-based pose-aware tokens instead of 1 token (ablation) | Pose-guided temporal framework · HanCo | J-PA: 6.03 -> 5.77 | Helped |
| use 6 finger-based tokens instead of 21 skeleton-based tokens (ablation) | Pose-guided temporal framework · HanCo | J-PA: 5.80 -> 5.77 | Helped |
| replace cross-attention with triplane encoding (ablation) | Pose-guided temporal framework · HanCo | J-PA: 5.95 -> 5.77 | Helped |
| replace single-plane encoding with triplane encoding (ablation) | Pose-guided temporal framework · HanCo | J-PA: 5.85 -> 5.77 | Helped |
| add 1D compression of the x-z and y-z plane features (ablation) | Triplane encoding · HanCo | J-PA: 5.80 -> 5.77 | Helped |
| fuse temporal features at multiple intermediate layers instead of only the final layer (ablation) | Pose-guided temporal framework · HanCo | J-PA: 6.01 -> 5.77 | Helped |
| use the proposed method at 64×64 rather than 32×32 input resolution | Pose-guided temporal framework · DexYCB | PA-MPJPE: 6.03 -> 5.72 | Helped |
Surprising findings
- Fusing temporal features at a middle layer outperforms fusing only at the final layer (J-PA 5.77 vs 6.01 on HanCo).
Limitations
- Does not explicitly model long-term hand motion cues across sequences, so hand shape consistency may degrade over extended sequences.
- Temporal window is short (T=6 frames with 5-frame interval chosen per dataset).
Experimental setup
Datasets: DexYCB HanCo H2O
Benchmarks: DexYCB HanCo H2O
Metrics: PA-MPJPE PA-MPVPE AUC F@5 F@15 MPJPE J-PA V-PA
Baselines: HMR + LR-ViT VIBE + LR-ViT METRO HandOccNet H2ONet SimpleHand Deformer TCMR HaMeR MobRecon Cho et al. HaMuCo
Builds on: ViTPose triplane representation MANO HMR VIBE
Closest papers by meaning
- SRHand: Super-Resolving Hand Images and 3D Shapes via View/Pose-aware Neural Image Representations and Explicit Meshes (NeurIPS 2025 · similarity 0.87)
- HOLD: Category-agnostic 3D Reconstruction of Interacting Hands and Objects from Video (CVPR 2024 · similarity 0.86)
- HandOS: 3D Hand Reconstruction in One Stage (CVPR 2025 · similarity 0.85)
- Fine-Grained Multi-View Hand Reconstruction Using Inverse Rendering (AAAI 2024 · similarity 0.85)
- Keypoint Fusion for RGB-D Based 3D Hand Pose Estimation (AAAI 2024 · similarity 0.85)
- EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.