Research map / Image generation / Fast diffusion sampling
SD4Match: Learning to Prompt Stable Diffusion Model for Semantic Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns universal, class-specific, or image-pair-conditioned prompt embeddings for a frozen Stable Diffusion UNet to improve semantic keypoint matching, reaching 75.5 PCK on SPair-71k.
Problem
Semantic keypoint correspondence across image pairs using features from Stable Diffusion, where prior work relied on handcrafted text prompts or implicit captions that may be suboptimal.
Approach
Treat the SD text prompt as a learnable embedding optimized directly with a correspondence cross-entropy loss while the UNet stays frozen, and add a conditional prompting module that conditions the prompt on DINOv2 local patch features from both images of the pair.
Key results
- SD4Match-Class and SD4Match-CPM reach 75.5 PCK@0.1 on SPair-71k, 12 points above the previous best SimSC-iBOT (63.5).
- SD4Match-Single improves over the zero-shot DIFT baseline on SPair-71k from 52.9 to 72.6 PCK@0.1.
- Generalization test trained on PF-Pascal gives SD4Match-CPM 52.1 PCK@0.05 on PF-Willow, but performance drops on SPair-71k relative to zero-shot baselines (27.2 vs 39.7 for DIFT at 0.05).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn a single universal prompt instead of using DIFT's textual-template prompt | Stable Diffusion semantic matching on SPair-71k · SPair-71k | PCK @ α=0.1: 37.2% | Helped |
| learn a category-specific prompt instead of a single universal prompt | SD4Match on SPair-71k · SPair-71k | PCK @ α=0.1: 72.6 -> 75.5 | Helped |
| use CPM conditioned on local features of the image pair instead of an individual-image global descriptor (ablation) | SD4Match-CPM on SPair-71k · SPair-71k | PCK @ α=0.1: 68.5 -> 75.5 | Helped |
| condition CPM on local feature patches rather than a global descriptor (ablation) | SD4Match-CPM on SPair-71k · SPair-71k | PCK @ α=0.1: 73.3 -> 75.5 | Helped |
| condition the prompt on the image pair rather than on an individual image (ablation) | SD4Match-CPM on SPair-71k · SPair-71k | PCK @ α=0.1: 70.8 -> 75.5 | Helped |
| add a global prompt to CPM (ablation) | SD4Match-CPM on SPair-71k · SPair-71k | PCK @ α=0.1: 74.6 -> 75.5 | Helped |
| add the patch-wise linear layer gn(·) to CPM (ablation) | SD4Match-CPM on SPair-71k · SPair-71k | PCK @ α=0.1: 74.0 -> 75.5 | Helped |
| replace the empty-string prompt with a category-template prompt | DIFT on SPair-71k · SPair-71k | PCK @ α=0.1: 50.7 -> 52.9 | Helped |
| tune SD4Match-CPM on PF-Pascal rather than use SD+DINO zero-shot | SD4Match-CPM · PF-Pascal test | PCK @ θimg, α=0.1: 85.7 -> 95.2 | Helped |
| tune SD4Match-CPM on PF-Pascal rather than use SD+DINO zero-shot | SD4Match-CPM · PF-Willow test | PCK @ θkps, α=0.1: 72.0 -> 80.4 | Helped |
| tune SD4Match-CPM on PF-Pascal rather than use SD+DINO zero-shot | SD4Match-CPM · SPair-71k | PCK @ θbbox, α=0.1: 59.3 -> 40.9 | Hurt |
| tune SD4Match-CPM on SPair-71k rather than PF-Pascal | SD4Match-CPM · SPair-71k | PCK @ θbbox, α=0.1: 40.9 -> 75.5 | Helped |
Surprising findings
- Using an empty text prompt already retains most of SD's matching ability (50.7 vs 52.9 for a category template on SPair-71k with DIFT).
- The optimal inference timestep (t=50) differs from the training timestep (t=261), with accuracy peaking at t=50 for the learned prompt.
Limitations
- Performance degrades on SPair-71k when prompts are tuned only on the smaller PF-Pascal dataset, indicating overfitting to small distributions.
- The class-specific variant requires known object category labels at inference.
Experimental setup
Datasets: SPair-71k PF-Pascal PF-Willow
Benchmarks: SPair-71k PF-Pascal PF-Willow
Metrics: PCK
Baselines: DIFT SD+DINO SimSC-iBOT CATs SCorrSAN Trans.Mat. SemiMatch PMNC DINOv2 DINO
Builds on: DIFT SimSC CoOp CoCoOp SD+DINO
Closest papers by meaning
- Diffusion Model for Dense Matching (ICLR 2024 · similarity 0.88)
- Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.87)
- CoMat: Aligning Text-to-Image Diffusion Model with Image-to-Text Concept Matching (NeurIPS 2024 · similarity 0.86)
- I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models (ICML 2025 · similarity 0.86)
- Text-Image Alignment for Diffusion-Based Perception (CVPR 2024 · similarity 0.86)
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.