Research map / Image generation / Text-to-image diffusion
SLiMe: Segment Like Me
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SLiMe, which optimizes Stable Diffusion text embeddings with cross- and weighted accumulated self-attention maps from a single annotated image to segment unseen images at any granularity.
Problem
Segmenting images at arbitrary part-level granularity with very few annotated samples, without category-specific GAN/diffusion training or large segmentation datasets.
Approach
Frame segmentation as one-shot optimization of Stable Diffusion text embeddings so that cross-attention maps, refined by a novel weighted accumulated self-attention (WAS-attention) map, highlight annotated regions; the optimized embeddings then segment unseen images without re-optimization.
Key results
- On PASCAL-Part car class in the 1-sample setting, SLiMe reaches 61.6 mIoU average versus 53.3 for SegGPT (supervised).
- On PASCAL-Part horse class in the 1-sample setting, SLiMe reaches 56.4 mIoU average versus 43.0 for SegDDPM and 40.3 for SegGPT.
- WAS-attention improves average mIoU by about 6.0 points over cross-attention alone on the car class (68.3 vs 62.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add WAS-attention to cross-attention (ablation) | SLiMe · PASCAL-Part car, 10-sample | mIoU: 62.7 ± 1.3 -> 68.3 ± 1.0 | Helped |
| change the initial prompt among part names, empty prompt, and repeated 'part' (ablation) | SLiMe · PASCAL-Part car, 10-sample | mIoU: 68.7 ± 0.4; 68.4 ± 1.2; 68.3 ± 1.0 | No clear effect |
| set SD regularization coefficient β to 0.5 rather than 0.005 (ablation) | SLiMe · PASCAL-Part car, 10-sample | mIoU: 68.3 ± 1.0 -> 5.3 ± 0.0 | Hurt |
| set SD regularization coefficient β to 0.05 rather than 0.005 (ablation) | SLiMe · PASCAL-Part car, 10-sample | mIoU: 68.3 ± 1.0 -> 65.4 ± 0.9 | Hurt |
| remove the SD regularization loss (ablation) | SLiMe · PASCAL-Part car, 10-sample | mIoU: 68.3 ± 1.0 -> 55.8 ± 1.6 | Hurt |
| add the WAS-attention MSE loss (α=1 rather than α=0) (ablation) | SLiMe · PASCAL-Part car, 10-sample | mIoU: 68.0 ± 0.3 -> 68.3 ± 1.0 | No clear effect |
| sample t_opt from [5,100] rather than [5,20] (ablation) | SLiMe · PASCAL-Part car, 10-sample | mIoU: 66.8 ± 0.2 -> 68.3 ± 1.0 | Helped |
| sample t_opt from [5,100] rather than [5,900] (ablation) | SLiMe · PASCAL-Part car, 10-sample | mIoU: 67.8 ± 1.0 -> 68.3 ± 1.0 | Helped |
| set t_test to 100 rather than 5 (ablation) | SLiMe · PASCAL-Part car, 10-sample | mIoU: 65.2 ± 1.4 -> 68.3 ± 1.0 | Helped |
| set t_test to 100 rather than 20 (ablation) | SLiMe · PASCAL-Part car, 10-sample | mIoU: 65.9 ± 0.7 -> 68.3 ± 1.0 | Helped |
| set learning rate to 0.1 rather than 1 (ablation) | SLiMe · PASCAL-Part car, 10-sample | mIoU: 15 ± 7.4 -> 68.3 ± 1.0 | Helped |
| set learning rate to 0.1 rather than 0.01 (ablation) | SLiMe · PASCAL-Part car, 10-sample | mIoU: 67.8 ± 0.3 -> 68.3 ± 1.0 | No clear effect |
Surprising findings
- Optimizing the text prompt content matters little: empty or generic 'part' prompts give nearly the same mIoU as part-name prompts.
- Using only cross-attention maps from the first 7 UNet layers yields very poor segmentation (22.2 mIoU), while middle layers 8-12 are needed.
Limitations
- Noisy or missed segmentations when the target region is tiny (e.g., necklace), due to low attention-map resolution.
- Limited to images and not yet applicable to 3D or video.
- Supports up to 77 classes due to SD's text token limit.
Experimental setup
Datasets: PASCAL-Part CelebAMask-HQ
Benchmarks: PASCAL-Part CelebAMask-HQ
Metrics: mIoU
Baselines: ReGAN SegDDPM SegGPT CNN CNN+CRF Shape+Appearance
Builds on: Stable Diffusion Prompt-to-Prompt cross-attention control DiffSeg SegDDPM
Closest papers by meaning
- Repurposing Stable Diffusion Attention for Training-Free Unsupervised Interactive Segmentation (CVPR 2025 · similarity 0.86)
- EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models (ICLR 2024 · similarity 0.86)
- Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.86)
- Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.85)
- Diffuse Attend and Segment: Unsupervised Zero-Shot Segmentation using Stable Diffusion (CVPR 2024 · similarity 0.85)
- SeeDiff: Off-the-Shelf Seeded Mask Generation from Diffusion Models (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.