Research map / Large language models / Language, safety and interpretability
LocLLM: Exploiting Generalizable Human Keypoint Localization via Large Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LocLLM, a vicuna-7B-based multimodal LLM that localizes human keypoints from images and text descriptions of keypoints, enabling generalization to unseen keypoints and cross-dataset settings.
Problem
Human keypoint localization models are restricted to keypoint priors encoded in architecture and training data, limiting generalization to unseen poses, datasets, and novel keypoint types.
Approach
Reformulates keypoint localization as visual question answering: an LLM receives the image plus a natural-language description of the target keypoint and outputs its coordinates as decimal strings, with LoRA-tuned visual encoder and LLM and a linear projector trained on localization-based instruction conversations.
Key results
- On COCO Keypoint val, LocLLM reaches 77.4 AP, comparable to ViTPose (77.4) and above the CLIP baseline (73.1).
- Cross-dataset on MPII, LocLLM reaches 33.4 PCKh@0.1, 7.8 above ViTPose (25.8); on Human-Art it reaches 64.8 AP vs 53.8 for ViTPose.
- When 4 keypoints are removed from training, LocLLM scores 67.1 AP on COCO versus 43.1 for the CLIP baseline; on MPII unseen Pelvis it reaches 43.6 vs 1.7.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add keypoint descriptions (ablation) | LocLLM · COCO Keypoint val | AP: 70.8 -> 72.4 | Helped |
| use decimal strings instead of location tokens (ablation) | LocLLM · COCO Keypoint val | AP: 67.2 -> 72.4 | Helped |
| use multi-round rather than single-round conversations (ablation) | LocLLM · COCO Keypoint val | AP: 68.9 -> 72.4 | Helped |
| add learnable modules to both visual encoder and LLM (ablation) | LocLLM · COCO Keypoint val | AP: 70.3 -> 72.4 | Helped |
| skip first-stage image-text alignment training (ablation) | LocLLM · COCO Keypoint val | AP: 72.4 -> 70.6 | Hurt |
| use LocLLM instead of ViTPose | keypoint localization models trained on COCO · Human-Art | AP: 53.8 -> 64.8 | Helped |
| use LocLLM instead of ViTPose | keypoint localization models trained on COCO · MPII cross-dataset, Mean0.1 | PCKh@0.1: 25.8 -> 33.4 | Helped |
| use LocLLM instead of RLE | monocular 3D keypoint localization · Human3.6M | MPJPE (Avg): 48.6 -> 46.6 | Helped |
| remove 4 keypoint types from training (ablation) | LocLLM · COCO Keypoint val | AP: 72.4 -> 67.1 | Hurt |
| use LocLLM instead of the CLIP baseline | novel-keypoint localization · MPII unseen keypoints (Pelvis) | Accuracy: 1.7 -> 43.6 | Helped |
Surprising findings
- The CLIP-based baseline with text-conditioned features fails badly on novel keypoints, matching them to similar training keypoints (e.g., pelvis to right hip).
Limitations
- Relies on accurate textual keypoint descriptions, so keypoints hard to describe in language (e.g., facial landmarks) are not handled well.
- The large LLM requires considerable GPU resources and reduces inference efficiency.
- On 3D Human3.6M, LocLLM (46.6 average) is worse than the best regression baseline Sun et al. (49.6) in the table's reported numbers.
Experimental setup
Datasets: CC3M COCO Keypoint MPII Human-Art Human3.6M
Benchmarks: COCO Keypoint Human3.6M Human-Art MPII
Metrics: AP AR PCKh@0.5 PCKh@0.1 MPJPE
Baselines: SimplePose HRNet SimCC ViTPose DeepPose RLE CLIP baseline CLAMP
Builds on: LLaVA LoRA DINOv2 Vicuna RLE
Closest papers by meaning
- KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension (NeurIPS 2024 · similarity 0.87)
- Text2Loc: 3D Point Cloud Localization from Natural Language (CVPR 2024 · similarity 0.85)
- Text to Point Cloud Localization with Multi-Level Negative Contrastive Learning (AAAI 2025 · similarity 0.85)
- Omnipotent Distillation with LLMs for Weakly-Supervised Natural Language Video Localization: When Divergence Meets Consistency (AAAI 2024 · similarity 0.84)
- Pixel-Aligned Language Model (CVPR 2024 · similarity 0.84)
- Mechanisms of Object Localization in Vision-Language Models (CVPR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.