Research map / Image generation / Text-to-image diffusion

SeeDiff: Off-the-Shelf Seeded Mask Generation from Diffusion Models

AAAI 2025 · Image generation › Text-to-image diffusion · approach: Multi-subject personalization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SeeDiff, which extracts masks from Stable Diffusion 2-base attention maps by seeded region expansion (cross-attention seeds, multi-scale self-attention expansion, background-mask refinement) without training or prompt tuning.

Problem

Generating pixel-level annotation masks for synthetic images from text-to-image diffusion models usually requires a pre-trained segmentation network, prompt tuning, or extra training, and cross-attention maps alone give coarse masks.

Approach

Treat cross-attention maps as seeds and expand them iteratively across resolutions using self-attention maps, like classical seeded segmentation, then refine the mask by subtracting a background mask obtained the same way from inverted seeds.

Key results

• On VOC 2012 val with Mask2Former Swin-B trained on 60k synthetic images, SeeDiff reaches 79.6 mIoU vs 70.6 for DiffuMask.
• Fine-tuning with 5k real images raises SeeDiff (Swin-B) to 88.8 mIoU vs 84.9 for DiffuMask fine-tuned.
• On Cityscapes (100k synthetic images, Swin-B), SeeDiff gets 74.3 Human and 89.8 Vehicle IoU vs 72.1 and 87.0 for DiffuMask.

Measured effects reported in the paper

Surprising findings

• Higher-resolution cross-attention maps lose objectness, so the 16x16 CA map works best as seeds (65.9 vs 64.3 and 58.2 mIoU for 32 and 64).

Limitations

• Relies on seeds from cross-attention, which can point to background near boundaries or be sparse for non-homogeneous objects.
• Background refinement assumes uniform backgrounds from simple text prompts.
• Evaluated mainly on VOC 2012 and two Cityscapes categories.

Experimental setup

Datasets: Pascal VOC 2012 Cityscapes

Benchmarks: Pascal VOC 2012 val Cityscapes val

Metrics: mIoU IoU

Baselines: DiffuMask Dataset Diffusion Mask2Former trained on real VOC data

Builds on: DiffuMask Dataset Diffusion Stable Diffusion

Closest papers by meaning

• Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models (CVPR 2024 · similarity 0.88)
• Explore In-Context Segmentation via Latent Diffusion Models (AAAI 2025 · similarity 0.87)
• Prompting Diffusion Models for Zero-Shot Instance Segmentation (NeurIPS 2026 · similarity 0.87)
• $\mathbf{D^{3}S^{2}}$: Diffusion-Guided Dataset Distillation for Semantic Segmentation (NeurIPS 2026 · similarity 0.87)
• Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.87)
• Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation (ICML 2025 · similarity 0.87)