Research map / Detection and segmentation / Open-set and few-shot detection
Seeing Beyond the Crop: Using Language Priors for Out-of-Bounding Box Keypoint Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TokenCLIPose, a regression-based top-down pose estimator that predicts keypoints outside the bounding box (e.g., hockey/lacrosse sticks) by initializing learnable keypoint tokens from CLIP text embeddings and fusing them with image and location tokens in a transformer decoder.
Problem
Top-down pose estimators crop tight boxes around people, so keypoints of interacting objects (sticks) lying outside the box are missed, while enlarging boxes adds clutter; occluded keypoints within boxes are also hard to localize.
Approach
Treat out-of-box extension keypoints as unseen keypoints and inject their semantics via CLIP text tokens used as learnable token initializations, combined with image and coarse location tokens in a self-attention transformer decoder supervised by RLE loss, instead of contrastively aligning image features to text.
Key results
- On the curated ice hockey dataset, TokenCLIPose (HRNet-W48) reaches 76.28 mean PCKh@0.5, a 4.36% gain over the best prior top-down method (ViTPose 71.92).
- Zero-shot transfer to the Lacrosse dataset yields 78.61 mean PCKh@0.5, 2.35% above ViTPose (76.26).
- On CrowdPose, TokenCLIPose reaches 76.2 AP at 384x288, 3.8% above HRFormer-B (72.4) and 4.8 AP above LAMP (71.4 at 512x512).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TokenCLIPose with HRNet-W48 instead of ViTPose | Ice hockey pose estimation · Ice hockey | PCKh@0.5 Mean: 4.36% | Helped |
| transfer TokenCLIPose zero-shot from ice hockey to Lacrosse | TokenCLIPose · Lacrosse | PCKh@0.5 Mean: 2.35% | Helped |
| use TokenCLIPose instead of the strongest listed top-down baseline | CrowdPose · CrowdPose test | AP: 3.8% | Helped |
| add text tokens (ablation) | TokenCLIPose · Ice hockey | Mean: 3.35% | Helped |
| add location tokens (ablation) | TokenCLIPose · Ice hockey | Mean: 0.45% | Helped |
| replace intention attention with self-attention (ablation) | TokenCLIPose decoder · Ice hockey | Mean: 75.14 -> 76.28 | Helped |
| replace self-attention with cross-attention (ablation) | TokenCLIPose decoder · Ice hockey | Mean: +0.03 | No clear effect |
| initialize text tokens from CLIP prompts instead of random initialization (ablation) | TokenCLIPose · Ice hockey | Mean: 2.31% | Helped |
| use prompt ensemble instead of a single prompt (ablation) | TokenCLIPose · Ice hockey | Mean: 1.04% | Helped |
| replace Faster-RCNN boxes with ground-truth boxes (ablation) | TokenCLIPose · Ice hockey | Mean: 2.47% | Helped |
Surprising findings
- Self-attention over all tokens performed about the same as separate self- and cross-attention per modality (76.28 vs 76.31).
- Location tokens added only a small gain (0.45%) while text tokens gave a 3.35% gain.
Limitations
- The paper's checklist says no limitations are discussed; the authors acknowledge the datasets are small (Lacrosse has only 300 annotations from one video).
- Datasets are not publicly released, limiting reproducibility and external validation.
- Performance depends on bounding box quality, with ground-truth boxes adding 2.47% gain over Faster R-CNN boxes.
Experimental setup
Datasets: ice hockey dataset (curated, NHL broadcast videos) Lacrosse dataset (curated, 300 annotations) CrowdPose
Benchmarks: ice hockey dataset (PCKh@0.5) Lacrosse dataset zero-shot (PCKh@0.5) CrowdPose (AP)
Metrics: PCKh@0.5 AP AP50 AP75 APE APM APH
Baselines: SimpleBaseline MSPN HRNet-W48 TokenPose-L/D24 ViTPose Mask-RCNN AlphaPose CrowdPose Hourglass-104 KAPAO-L Transpose-H HRFormer-B
Builds on: CLIP RLE (residual log-likelihood estimation) TokenPose LAMP RealNVP
Closest papers by meaning
- Optimizing Human Pose Estimation Through Focused Human and Joint Regions (AAAI 2025 · similarity 0.85)
- ProbPose: A Probabilistic Approach to 2D Human Pose Estimation (CVPR 2025 · similarity 0.84)
- UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editing (CVPR 2025 · similarity 0.84)
- ShotVL: Human-Centric Highlight Frame Retrieval via Language Queries (AAAI 2025 · similarity 0.84)
- FreeMan: Towards Benchmarking 3D Human Pose Estimation under Real-World Conditions (CVPR 2024 · similarity 0.83)
- SHaRPose: Sparse High-Resolution Representation for Human Pose Estimation (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.