Research map / Vision-language and multimodal / Multimodal LLMs
KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Semantic Keypoint Comprehension and KptLLM, a LLaVA-based MLLM with a prompt feature extractor and identify-then-detect chain-of-thought that handles keypoint semantics, visual-prompt and textual-prompt keypoint detection.
Problem
MLLMs struggle with pixel-level semantic detail such as object keypoints; existing keypoint detectors rely on visual pattern fitting, lack semantic understanding of keypoints, and have fixed input-output formats.
Approach
A unified MLLM that first identifies keypoint semantics via the LLM and then localizes them (identify-then-detect), using a SAM-inspired prompt encoder and a trainable cross-attention prompt feature extractor over support image features, with a special <keypoint> token decoded by an FFN head into coordinates.
Key results
- Visual prompt 1-shot mean PCK on MP-100 reaches 87.10 vs 85.31 for CapeFormer; 5-shot mean 90.19 vs 89.30.
- Textual prompt AP-10K (Canidae->Felidae) AP 70.2 vs 48.4 for CLAMP.
- Keypoint semantic understanding accuracy 83% vs 72% for LoRA-finetuned LLaVA and 3% for original LLaVA on MP-100 Split-1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune LLaVA on MP-100 using LoRA | LLaVA · MP-100 Split-1 | accuracy: 3% -> 72% | Helped |
| Use KptLLM instead of fine-tuned LLaVA | Keypoint semantic understanding · MP-100 Split-1 | accuracy: 72% vs 83% | Helped |
| Use KptLLM instead of CapeFormer | Visual prompt-based detection · MP-100 1-shot, mean over five splits | PCK: 85.31 -> 87.10 | Helped |
| Use KptLLM instead of CapeFormer | Visual prompt-based detection · MP-100 5-shot, mean over five splits | PCK: 89.30 -> 90.19 | Helped |
| Use KptLLM instead of CLAMP | Textual prompt-based detection · AP-10K, Bovidae to Canidae | AP: 15.3 AP improvement | Helped |
| Use KptLLM instead of CLAMP | Textual prompt-based detection · AP-10K, Canidae to Felidae | AP: 21.8 AP increase | Helped |
| Add the identify-then-detect strategy (ablation) | KptLLM · MP-100 Split-1, 1-shot | PCK: 87.68 -> 91.66 | Helped |
| Use the trainable prompt feature extractor instead of average pooling (ablation) | KptLLM · MP-100 Split-1, 1-shot | PCK: 89.78 -> 91.66 | Helped |
| Add a textual prompt alongside the visual prompt (ablation) | KptLLM · MP-100 Split-1, 1-shot | PCK: 91.66 -> 92.18 | Helped |
Surprising findings
- Original LLaVA achieves only 3% keypoint semantic understanding accuracy, showing general MLLMs largely fail at keypoint semantics.
Limitations
- Large model size and computational cost typical of MLLMs.
- Limited diversity of object and keypoint categories in MP-100 and AP-10K.
- Keypoint output via special token decoded by FFN rather than text coordinates, which limits user interaction.
Experimental setup
Datasets: MP-100 AP-10K X-Pose keypoint semantic labels
Benchmarks: MP-100 AP-10K
Metrics: accuracy PCK@0.2 AP AP50 AP75 APM APL AR
Baselines: LLaVA ProtoNet MAML Finetune POMNet CapeFormer SimpleBaseline CLAMP
Builds on: LLaVA SAM POMNet CLAMP LoRA
Closest papers by meaning
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.87)
- LocLLM: Exploiting Generalizable Human Keypoint Localization via Large Language Model (CVPR 2024 · similarity 0.87)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.86)
- VisMin: Visual Minimal-Change Understanding (NeurIPS 2024 · similarity 0.86)
- UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning (NeurIPS 2025 · similarity 0.86)
- ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.