Research map / Vision-language and multimodal / CLIP and zero-shot
Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a frozen Stable Diffusion v2.1 as a backbone feature extractor for zero-shot sketch-based image retrieval, learning only visual and textual prompts with a triplet loss.
Problem
Zero-shot sketch-based image retrieval (ZS-SBIR) and fine-grained ZS-SBIR need backbones that bridge the sketch-photo domain gap, but existing backbones are discriminatively pre-trained CNNs/ViTs or CLIP, and the potential of text-to-image diffusion features for this task is unexplored.
Approach
Use intermediate UNet upsampling features of a frozen text-to-image diffusion model as cross-modal, shape-biased embeddings, and adapt them via learnable pixel-space border visual prompts and a learnable continuous textual prompt, plus feature ensembling over noise samples and layer/time-step selection.
Key results
- On category-level ZS-SBIR, the method reaches mAP@200 of 0.746 on Sketchy, versus 0.723 for the strongest prior method ZS-LVM.
- On cross-category ZS-FG-SBIR on Sketchy, Acc.@1 is 31.94 versus 28.68 for CC-LVM.
- Pilot study: frozen VGG-16 gets 49.14% SBIR accuracy on the cue-conflict gallery versus 72.82% for the SD generative backbone.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use frozen Stable Diffusion features with learned visual and textual prompts | ZS-SBIR · Sketchy | mAP@200: 0.723 -> 0.746 | Helped |
| Use frozen Stable Diffusion features with learned visual and textual prompts | ZS-SBIR · TU-Berlin | mAP@all: 0.651 -> 0.680 | Helped |
| Use frozen Stable Diffusion features with learned visual and textual prompts | ZS-SBIR · Quick, Draw! | mAP@all: 0.202 -> 0.231 | Helped |
| Use the proposed frozen diffusion feature pipeline | ZS-FG-SBIR · Sketchy | Acc.@1: 28.68 -> 31.94 | Helped |
| Add a learned visual prompt (ablation) | Stable Diffusion ZS-SBIR · Sketchy | mAP@200: 0.513 -> 0.746 | Helped |
| Add a learned visual prompt (ablation) | Stable Diffusion ZS-FG-SBIR · Sketchy | Acc.@1: 17.66 -> 31.94 | Helped |
| Add a learned textual prompt (ablation) | Stable Diffusion ZS-SBIR · Sketchy | mAP@200: 0.695 -> 0.746 | Helped |
| Add a learned textual prompt (ablation) | Stable Diffusion ZS-FG-SBIR · Sketchy | Acc.@1: 22.71 -> 31.94 | Helped |
| Use feature ensembling (ablation) | Stable Diffusion ZS-SBIR · Sketchy | mAP@200: 0.725 -> 0.746 | Helped |
| Use feature ensembling (ablation) | Stable Diffusion ZS-FG-SBIR · Sketchy | Acc.@1: 29.47 -> 31.94 | Helped |
| Use upsampling features rather than downsampling features (ablation) | Stable Diffusion ZS-SBIR · Sketchy | mAP@200: 0.365 -> 0.746 | Helped |
| Use upsampling features rather than downsampling features (ablation) | Stable Diffusion ZS-FG-SBIR · Sketchy | Acc.@1: 18.22 -> 31.94 | Helped |
Surprising findings
- Fine-tuning the full SD UNet with triplet loss collapses retrieval (mAP@200 0.115 on Sketchy), showing generative knowledge is lost under discriminative fine-tuning.
- Sketch-photo feature maps from SD's intermediate upsampling layers share semantic correspondence despite the modality gap.
Limitations
- Relies on a single backbone (Stable Diffusion v2.1) and a fixed prompt-based setup; evaluated mainly on Sketchy, TU-Berlin and Quick, Draw!
- Sensitive to the choice of layers and time-step, which required grid search on the test dataset (Sketchy)
- Feature extraction requires multiple noise samples (six) per image, adding inference cost
Experimental setup
Datasets: Sketchy Quick, Draw! TU-Berlin cue-conflict FS-COCO Sketchy-COCO
Benchmarks: Sketchy TU-Berlin Quick, Draw! FS-COCO Sketchy-COCO
Metrics: mAP@200 P@200 mAP@all P@100 Acc.@1 Acc.@5
Baselines: ZS-CVAE ZS-CAAE ZS-CCGAN ZS-GRL ZS-SAKE ZS-IIAE ZS-Sketch3T ZS-LVM
Builds on: Stable Diffusion ZS-LVM visual prompting triplet loss DIFT
Closest papers by meaning
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
- SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models (CVPR 2025 · similarity 0.89)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- The Hidden Language of Diffusion Models (ICLR 2024 · similarity 0.88)
- Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function (NeurIPS 2024 · similarity 0.88)
- It's All About Your Sketch: Democratising Sketch Control in Diffusion Models (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.