Research map / 3D vision / Human pose estimation
Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hourglass Tokenizer (HoT), a plug-and-play framework that prunes redundant pose tokens with a parameter-free density-peaks clustering module (TPC) and recovers full-length tokens with a lightweight cross-attention module (TRA), cutting FLOPs of video pose transformers for 3D human pose estimation.
Problem
Video pose transformers for 3D human pose estimation have quadratic self-attention cost in the number of frames, making them impractical on resource-constrained devices.
Approach
Prune pose tokens of redundant frames in intermediate transformer blocks using density-peaks clustering (DPC-kNN) to select semantically diverse cluster-center tokens, then recover full-length tokens at the end with a token recovering cross-attention, giving an hourglass token-count profile.
Key results
- On Human3.6M, HoT w. MotionBERT cuts FLOPs by 51.8% (131.09G to 63.21G) with MPJPE unchanged at 39.8mm.
- HoT w. MixSTE reduces FLOPs by 39.6% (277.25G to 167.52G) with MPJPE 41.0mm vs 40.9mm for the baseline.
- TPC w. MotionBERT (n=2, f=121) reaches 39.0mm MPJPE vs 39.2mm baseline while cutting FLOPs by 30.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TPC pruning to MotionBERT in the seq2frame pipeline | MotionBERT · Human3.6M | MPJPE (mm): 39.5mm to 39.2mm | Helped |
| Add TPC pruning and TRA recovery to MotionBERT | MotionBERT, seq2seq · Human3.6M | FLOPs (G): 131.09 -> 63.21 | Helped |
| Add TPC pruning and TRA recovery to MotionBERT | MotionBERT, seq2seq · Human3.6M | MPJPE (mm): 39.8 -> 39.8 | No clear effect |
| Add TPC pruning and TRA recovery to MixSTE | MixSTE, seq2seq · Human3.6M | FLOPs (G): 277.25 -> 167.52 | Helped |
| Add TPC pruning and TRA recovery to MixSTE | MixSTE, seq2seq · Human3.6M | MPJPE (mm): 40.9 -> 41.0 | Hurt |
| Use f=9 representative tokens instead of the baseline configuration (ablation) | HoT w. MixSTE · Human3.6M, seq2seq | MPJPE (mm): 40.9 -> 43.5 | Hurt |
| Use f=16 representative tokens instead of the baseline configuration (ablation) | HoT w. MixSTE · Human3.6M, seq2seq | MPJPE (mm): 40.9 -> 42.2 | Hurt |
| Use f=61 representative tokens instead of the baseline configuration (ablation) | HoT w. MixSTE · Human3.6M, seq2seq | MPJPE (mm): 40.9 -> 41.2 | Hurt |
| Use f=135 representative tokens instead of f=81 (ablation) | HoT w. MixSTE · Human3.6M, seq2seq | MPJPE (mm): 41.0 -> 41.3 | Hurt |
| Prune after block n=2 (ablation) | TPC w. MixSTE, seq2frame · Human3.6M | MPJPE (mm): 40.7 -> 40.7 | No clear effect |
| Prune after block n=7 (ablation) | TPC w. MixSTE, seq2frame · Human3.6M | MPJPE (mm): 40.7 -> 39.9 | Helped |
| Replace TPC with uniform sampling (ablation) | MixSTE, seq2seq · Human3.6M | MPJPE of selected frames (mm): 40.2mm vs. 41.4mm | Hurt |
Surprising findings
- Pruning without retraining (TPC on MixSTE at block 2 or 3) keeps accuracy while cutting FLOPs, suggesting deep transformer blocks contain redundancy.
- Attention-score pruning tends to select tokens concentrated in the left half of a sequence, lacking diversity.
Limitations
- HoT with token recovering is slightly less accurate than TPC alone in the seq2frame setting, since recovery uses limited information.
- Reported gains depend on the specific VPT backbone and chosen block index and token count hyperparameters.
Experimental setup
Datasets: Human3.6M MPI-INF-3DHP
Benchmarks: Human3.6M MPI-INF-3DHP
Metrics: MPJPE PCK AUC FLOPs FPS parameters
Baselines: MHFormer MixSTE MotionBERT PoseFormer Strided Transformer P-STMO STCFormer VPose UGCN Anatomy3D DeciWatch Uniform Sampling
Builds on: MixSTE MotionBERT MHFormer DPC-kNN DynamicViT
Closest papers by meaning
- TokenHMR: Advancing Human Mesh Recovery with a Tokenized Pose Representation (CVPR 2024 · similarity 0.88)
- Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction (CVPR 2025 · similarity 0.86)
- SDPose: Tokenized Pose Estimation via Circulation-Guide Self-Distillation (CVPR 2024 · similarity 0.85)
- Semantics-aware Test-time Adaptation for 3D Human Pose Estimation (ICML 2025 · similarity 0.85)
- Exploring Token Pruning in Vision State Space Models (NeurIPS 2024 · similarity 0.85)
- Optimizing Human Pose Estimation Through Focused Human and Joint Regions (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.