Research map / 3D vision / Human pose estimation
CapeX: Category-Agnostic Pose Estimation from Textual Point Explanation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CapeX, a category-agnostic pose estimator that replaces visual support images with text-described keypoint graphs (frozen gte-base-v1.5 text encoder), and reports new 1-shot SOTA on MP-100.
Problem
Category-agnostic pose estimation (CAPE) normally needs annotated support images to define keypoints on novel categories; visual support matching is cumbersome and ambiguous across instances.
Approach
Represent the support keypoint definitions as a pose graph whose nodes carry open-vocabulary text descriptions embedded by a frozen text encoder, matched against query image features via similarity heatmaps and a graph transformer decoder, so no support image is needed.
Key results
- Under 1-shot on MP-100, CapeX averages 91.50 PCK@0.2 versus 90.43 for Pose Anything-S, a 1.07-point gain and new SOTA.
- Frozen gte-base-v1.5 text backbone (91.50) outperforms fine-tuning it (90.98) and the CLIP text variants (about 90.15-90.37).
- Masking query images shows CapeX degrades similarly to competitors at 50% occlusion, where performance drops to match them.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace visual support-image prompts with a text-graph prompt | CapeX · MP-100, 1-shot, five-split average | PCK@0.2: 90.43 -> 91.50 | Helped |
| Use a graph transformer decoder instead of the MLP transformer decoder (ablation) | CapeX with frozen gte-base-v1.5 · MP-100, 1-shot, five-split average | PCK@0.2: 91.22 -> 91.50 | Helped |
| Fine-tune the gte-base-v1.5 text backbone instead of freezing it (ablation) | CapeX with graph transformer decoder · MP-100, 1-shot, five-split average | PCK@0.2: 91.50 -> 90.98 | Hurt |
| Use frozen CLIP ViT-B/32 instead of frozen gte-base-v1.5 (ablation) | CapeX with graph transformer decoder · MP-100, 1-shot, five-split average | PCK@0.2: 91.50 -> 90.37 | Hurt |
| Fine-tune CLIP ViT-B/32 instead of freezing it (ablation) | CapeX with graph transformer decoder · MP-100, 1-shot, five-split average | PCK@0.2: 90.37 -> 90.15 | Hurt |
Surprising findings
- Fine-tuning the text backbone hurt performance and greatly increased memory and time, which the authors attribute to overfitting.
- At 50% query masking, the text-only approach loses its advantage and matches image-support competitors.
Limitations
- Fails on categories outside the training distribution and on novel text-graphs, such as the plane example.
- Trained on a relatively small dataset for a short period compared with large vision-language foundation models.
- Struggles with semantically challenging or vastly new keypoint descriptions, e.g., a horse 'seat'.
Experimental setup
Datasets: MP-100
Benchmarks: MP-100
Metrics: PCK@0.2
Baselines: ProtoNet MAML Fine-tuned POMNet CapeFormer CapeFormer-S Pose Anything-S
Builds on: Pose Anything CapeFormer POMNet
Closest papers by meaning
- Meta-Point Learning and Refining for Category-Agnostic Pose Estimation (CVPR 2024 · similarity 0.92)
- ESCAPE: Encoding Super-keypoints for Category-Agnostic Pose Estimation (CVPR 2024 · similarity 0.90)
- Open-Vocabulary Object 6D Pose Estimation (CVPR 2024 · similarity 0.87)
- Universal Features Guided Zero-Shot Category-Level Object Pose Estimation (AAAI 2025 · similarity 0.86)
- Rethinking Correspondence-based Category-Level Object Pose Estimation (CVPR 2025 · similarity 0.86)
- SecondPose: SE(3)-Consistent Dual-Stream Feature Fusion for Category-Level Pose Estimation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.