Research map / Image generation / Text-to-image diffusion
Where's Waldo: Diffusion Features For Personalized Segmentation and Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PDM, a training-free method that matches intermediate Stable Diffusion (SDXL-Turbo) features (self-attention Q/K appearance plus cross-attention semantic maps) for personalized retrieval and segmentation, and introduces PerMIR/PerMIS multi-instance benchmarks built from BURST.
Problem
Personalized retrieval and segmentation must locate a specific reference instance, but self-supervised foundation models (DINOv2, SAM, DIFT) fail when other instances of the same class are present, and existing benchmarks rarely contain such hard negatives.
Approach
Use features from the last U-Net layer of a pre-trained text-to-image diffusion model after image inversion: self-attention Q/K features for appearance, cross-attention features with the class-name token for semantics, fused into one similarity map, with no training.
Key results
- On PerMIS (image) segmentation, PDM reaches 42.3 mIoU vs 20.2 for DINOv2 and 21.9 for DIFT, and 49.7 mIoU when combined with PerSAM.
- On PerMIR retrieval, PDM achieves 73.0 mAP vs 29.7 for DINOv2 and 26.7 for OpenCLIP; DINOv2 + PDM reaches 80.4 mAP on ROxford-Medium.
- On RParis and ROxford, SuperGlobal + PDM re-ranking reaches 94.0 and 91.2 mAP (Medium), slightly above supervised SuperGlobal.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an object mask instead of a class name (ablation) | PDM · PerMIR | mAP: 73.0 -> 76.2 | Helped |
| Combine appearance and semantic maps by averaging them instead of using only the appearance map (ablation) | PDM · PerMIR | mAP: 42.3 -> 73.0 | Helped |
| Combine appearance and semantic maps by averaging them instead of using only the semantic map (ablation) | PDM · PerMIR | mAP: 32.9 -> 73.0 | Helped |
| Apply PerSAM with PDM features | PDM · PerSeg | mIoU: 95.4 -> 97.4 | Helped |
| Apply PerSAM with PDM features | PDM · PerMIS-Image | mIoU: 42.3 -> 49.7 | Helped |
| Apply PerSAM with PDM features | PDM · DAVIS | J & F: 75.8 -> 78.0 | Helped |
| Apply PerSAM with PDM features | PDM · PerMIS-Video | J & F: 75.1 -> 76.5 | Helped |
| Use PDM rather than DINOv2 for retrieval | retrieval methods · PerMIR | mAP: 29.7 -> 73.0 | Helped |
| Add PDM reranking to DINOv2 | DINOv2 · ROxford Hard | mAP: 54.0 -> 62.1 | Helped |
| Add PDM reranking to SuperGlobal | SuperGlobal · RParis Hard | mAP: 86.7 -> 86.8 | Helped |
| Add PDM reranking to OpenClip | OpenClip · ROxford Medium | mAP: 50.7 -> 70.1 | Hurt |
Surprising findings
- Using only a class-name prompt with diffusion features outperforms using a ground-truth object mask for personalized retrieval (73.0 vs 76.2 mAP reversed: mask-based gave 76.2 vs 73.0 with class name, so mask helped slightly).
- On DAVIS, PerSAM(SAM) and DINOv2 variants show that a diffusion-based fused map does not dominate all benchmarks; PDM is competitive rather than best on DAVIS J&F (75.8 vs 78.0 for PerSAM with PDM).
Limitations
- Requires image inversion for feature extraction, so performance depends on reconstruction quality.
- Inversion is the main computational bottleneck, though reduced with SDXL-Turbo.
- Does not beat all methods on DAVIS (e.g. PerSAM with PDM achieves higher J&F than PDM alone).
Experimental setup
Datasets: PerSeg DAVIS17 ROxford RParis BURST
Benchmarks: PerSeg PerMIS DAVIS17 ROxford RParis PerMIR
Metrics: mIoU bIoU J&F J F mAP
Baselines: DINOv2 PerSAM DIFT DiffSeg SEEM SegGPT MAST SFC MAE iBOT CLIP OpenCLIP
Builds on: Stable Diffusion DIFT PerSAM SAM BURST
Closest papers by meaning
- Mining your own secrets: Diffusion Classifier Scores for Continual Personalization of Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.89)
- Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025 · similarity 0.86)
- Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation (NeurIPS 2024 · similarity 0.86)
- PaRa: Personalizing Text-to-Image Diffusion via Parameter Rank Reduction (ICLR 2025 · similarity 0.85)
- Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers (CVPR 2024 · similarity 0.85)
- Personalized Preference Fine-tuning of Diffusion Models (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.