Research map / Detection and segmentation / Open-set and few-shot detection
Generalizable Object Keypoint Localization from Generative Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GenLoc, which localizes object keypoints by optimizing learnable keypoint condition embeddings and LoRA-tuned Stable Diffusion 1.5 U-Net features via a condition-guided correlation map (CKHE) and a heatmap expectation loss (HEL).
Problem
Generalizable object keypoint localization fails with limited keypoint training data, leading to poor cross-domain, few-shot and zero-shot generalization.
Approach
Instead of fine-tuning the diffusion U-Net to output heatmaps, it treats keypoint conditions as queries and computes a softmax over image tokens (CKHE) to form a correlation map, supervised by an integral-style expectation loss (HEL) that maximizes overlap with the target Gaussian heatmap.
Key results
- Achieves 80.1% AP on AP-10K, outperforming CLAMP-L (77.8%) by 2.3 points.
- Reaches 62.8% AP in 5-shot AP-10K setting, 9.5 points above ScarceNet.
- Zero-shot Bovidae-to-Canidae: 72.8% AP vs CLAMP 46.9% AP; 67.0% AP on Human-Art cross-domain vs 53.8% for ViTPose.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GenLoc instead of CLAMP-L | AP-10K animal keypoint localization · AP-10K | AP: 77.8 -> 80.1 | Helped |
| Use GenLoc instead of ViTPose | COCO human keypoint localization · COCO Keypoint | AP: 77.4 -> 77.6 | Helped |
| Use GenLoc instead of LocLLM | cross-domain human keypoint localization · Human-Art val (trained on COCO Keypoint) | AP: 64.8 -> 67.0 | Helped |
| Use GenLoc instead of ScarceNet | few-shot animal keypoint localization · AP-10K, 5 images/species | AP: 0.533 -> 0.628 | Helped |
| Use GenLoc instead of ScarceNet | few-shot animal keypoint localization · AP-10K, 10 images/species | AP: 0.597 -> 0.668 | Helped |
| Use GenLoc instead of ScarceNet | few-shot animal keypoint localization · AP-10K, 15 images/species | AP: 0.632 -> 0.688 | Helped |
| Use GenLoc instead of ScarceNet | few-shot animal keypoint localization · AP-10K, 20 images/species | AP: 0.654 -> 0.692 | Helped |
| Use GenLoc instead of ScarceNet | few-shot animal keypoint localization · AP-10K, 25 images/species | AP: 0.681 -> 0.711 | Helped |
| Use GenLoc instead of CLAMP | zero-shot animal keypoint localization · AP-10K, Bovidae to Canidae | AP: 46.9 -> 72.8 | Helped |
| Use GenLoc instead of CLAMP | zero-shot animal keypoint localization · AP-10K, Bovidae to Equidae | AP: 46.6 -> 76.0 | Helped |
| Use GenLoc instead of CLAMP | zero-shot animal keypoint localization · AP-10K, Bovidae to Felidae | AP: 28.7 -> 64.3 | Helped |
| Use GenLoc instead of CLAMP | zero-shot animal keypoint localization · AP-10K, Canidae to Felidae | AP: 48.4 -> 73.4 | Helped |
Surprising findings
- Optimizing only the condition embeddings (as in prior cross-attention methods) yields just 60.1% AP, showing LoRA tuning is needed for localization.
- Using clean images as input outperforms adding noise, contrary to typical diffusion feature extraction practice.
Limitations
- Relies on large pre-trained diffusion model, so compute and memory costs of inference are not analyzed.
- Evaluated only on COCO, Human-Art and AP-10K; no results on other keypoint domains such as hands or faces.
Experimental setup
Datasets: COCO Keypoint AP-10K Human-Art
Benchmarks: AP-10K COCO Keypoint Human-Art
Metrics: AP AP50 AP75 APM APL AR
Baselines: CLAMP ScarceNet HRNet ViTPose SimplePose RTMPose LocLLM UDA FixMatch FlexMatch RLE Hedlin et al. cross-attention baseline
Builds on: Stable Diffusion Marigold Integral Pose SLiMe CLAMP
Closest papers by meaning
- LocLLM: Exploiting Generalizable Human Keypoint Localization via Large Language Model (CVPR 2024 · similarity 0.83)
- Reloc3r: Large-Scale Training of Relative Camera Pose Regression for Generalizable, Fast, and Accurate Visual Localization (CVPR 2025 · similarity 0.83)
- Incremental Object Keypoint Learning (CVPR 2025 · similarity 0.82)
- Spatial-Aware Regression for Keypoint Localization (CVPR 2024 · similarity 0.82)
- Generative Region-Language Pretraining for Open-Ended Object Detection (CVPR 2024 · similarity 0.82)
- GPVK-VL: Geometry-Preserving Virtual Keyframes for Visual Localization under Large Viewpoint Changes (CVPR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.