Research map / Image generation / Text-to-image diffusion
Unsupervised Keypoints from Pretrained Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Optimizes text embeddings of Stable Diffusion 1.5 so cross-attention maps become localized Gaussians, yielding unsupervised keypoints that outperform prior methods on unaligned datasets.
Problem
Unsupervised keypoint/landmark learning lags supervised methods, especially on non-aligned, in-the-wild data, and existing methods rely on alignment priors or per-dataset training.
Approach
Instead of training a network, optimize a set of text-embedding tokens in a frozen text-to-image diffusion model so that their cross-attention maps become single-mode Gaussians at their maxima, with equivariance under small affine transforms; the localization objective also discourages tokens from collapsing onto the same location.
Key results
- On CUB non-aligned subsets, the method reaches 10.5/11.1/10.3 (CUB-001/002/003) and 5.4 on CUB-all, roughly half the error of prior unsupervised methods (e.g., Autolink 11.6 on CUB-all).
- On the wild CelebA setting (K=4), error is 5.24 versus 6.11 for the best prior (Autolink†); on Tai-Chi-HD it reaches 234.89 cumulative l2 versus 316.10 for Autolink†.
- On aligned CelebA, Human3.6M, and CUB-aligned the method is comparable but not best (e.g., 3.60 vs 3.19 for IMM on aligned CelebA K=10).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize keypoints using 100 training images | Our method · CelebA wild (K=8) | normalized ℓ2 error: 5.33 | Helped |
| Remove test-time ensembling (ablation) | Our method · CUB-all | normalized ℓ2 error: 5.4 -> 5.6 | Hurt |
| Remove furthest point sampling (ablation) | Our method · CUB-all | normalized ℓ2 error: 5.4 -> 6.4 | Hurt |
| Remove query Q upsampling (ablation) | Our method · CUB-all | normalized ℓ2 error: 5.4 -> 8.0 | Hurt |
| Remove equivariance loss (ablation) | Our method · CUB-all | normalized ℓ2 error: 5.4 -> 22.2 | Hurt |
| Use optimized diffusion text embeddings for keypoints instead of the strongest reported baseline | CelebA wild (K=4) · CelebA wild (K=4) | normalized ℓ2 error: 6.11 -> 5.24 | Helped |
| Use optimized diffusion text embeddings for keypoints instead of the strongest reported baseline | Our method · CUB-all | normalized ℓ2 error: 9.2 -> 5.4 | Helped |
| Use optimized diffusion text embeddings for keypoints instead of the strongest reported baseline | Our method · Tai-Chi-HD | cumulative ℓ2 error: 316.10 -> 234.89 | Helped |
Surprising findings
- Tokens optimized on CUB birds generalize to Tai-Chi-HD human poses at roughly state-of-the-art level.
- Optimizing on only 100 CelebA images still gives performance comparable to state of the art.
Limitations
- Aligned settings are not best; the method trails alignment-prior methods on CelebA aligned, CUB-aligned, and Human3.6M standard protocol.
- Test-time ensembling over augmentations increases inference cost linearly.
- Cross-dataset generalization is mixed (e.g., CelebA tokens on CUB work only when successful).
Experimental setup
Datasets: CelebA CUB-200-2011 Tai-Chi-HD DeepFashion Human3.6M
Benchmarks: CelebA CUB-200-2011 Tai-Chi-HD DeepFashion Human3.6M
Metrics: normalized l2 error (inter-ocular distance) mean l2 error cumulative l2 error PCK@6px relative l2 at 128x128
Baselines: Autolink Lorenz et al. LatentKeypointGAN IMM GANSeg Thewlis et al. Zhang et al. SCOPS
Builds on: Stable Diffusion Hedlin et al. (diffusion correspondence) Autolink equivariance loss
Closest papers by meaning
- Diffusion Bridge: Leveraging Diffusion Model to Reduce the Modality Gap Between Text and Vision for Zero-Shot Image Captioning (CVPR 2025 · similarity 0.86)
- Precise Parameter Localization for Textual Generation in Diffusion Models (ICLR 2025 · similarity 0.85)
- NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models (CVPR 2024 · similarity 0.85)
- UniMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation (ICML 2025 · similarity 0.85)
- Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models (NeurIPS 2025 · similarity 0.85)
- Unsupervised Region-Based Image Editing of Denoising Diffusion Models (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.