Research map / Image generation / Text-to-image diffusion
Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes joint attention in MM-DiT text-to-image models, identifies a semantic grounding expert layer (layer 9 of SD3) whose I2T attention yields zero-shot segmentation, and applies LoRA fine-tuning (MAGNET) with mask losses to improve segmentation and generation.
Problem
How and where the joint image-text attention in multi-modal diffusion transformers (MM-DiT) contributes to semantic grounding, and whether it can be used for open-vocabulary and unsupervised segmentation without training.
Approach
Identify a single MM-DiT block (the semantic grounding expert, layer 9 of SD3) whose image-to-text attention aligns text tokens with coherent image regions, use its I2T maps as zero-shot segmentation logits, and lightly fine-tune that layer with LoRA using mask-matching losses (MAGNET) to strengthen grouping and also improve generation.
Key results
- Zero-shot open-vocabulary segmentation with SD3 layer 9 I2T attention reaches 89.2 mIoU on VOC20 and 62.0 on COCO-Object; MAGNET with COCO training raises these to 89.8 and 62.9.
- On unsupervised segmentation, SD3 with MAGNET (COCO) reaches 53.5 mIoU on Stuff-27 and 45.4 on ADE, versus 44.2 and 37.7 for DiffSeg (SD1.5) on those sets.
- MAGNET (COCO) raises average CLIPScore from 27.14 (baseline) to 27.28 across Pick-a-Pic, COCO and SA-1B prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MAGNET trained on COCO to Seg4Diff | SD3 Seg4Diff · PascalVOC val | mIoU: 89.2 -> 89.8 | Helped |
| Add MAGNET trained on COCO to Seg4Diff | SD3 Seg4Diff · Pascal Context-59 val | mIoU: 49.0 -> 51.2 | Helped |
| Add MAGNET trained on COCO to Seg4Diff | SD3 Seg4Diff · Cityscapes val | mIoU: 26.5 -> 26.0 | Hurt |
| Add MAGNET trained on COCO to Seg4Diff | SD3 Seg4Diff · MS-COCO | CLIPScore: 26.0638 -> 26.2319 | Helped |
| Add MAGNET trained on COCO to Seg4Diff | SD3 Seg4Diff · T2I-CompBench++ | 2D object relationships: 0.2435 -> 0.2301 | Hurt |
| Use softmax-normalized attention scores instead of raw pre-softmax scores (ablation) | Seg4Diff attention mask · PascalVOC | mIoU: 83.2 -> 89.0 | Helped |
| Use softmax-normalized attention scores instead of raw pre-softmax scores (ablation) | Seg4Diff attention mask · COCO-Object | mIoU: 55.1 -> 61.8 | Helped |
| Add min-max normalization to softmax attention scores (ablation) | Seg4Diff attention mask · PascalVOC | mIoU: 89.0 -> 88.3 | Hurt |
| Add min-max normalization to softmax attention scores (ablation) | Seg4Diff attention mask · COCO-Object | mIoU: 61.8 -> 57.0 | Hurt |
| Set input-image timestep to t=8 of 28 rather than other tested timesteps (ablation) | Seg4Diff segmentation inference · PascalVOC and COCO-Object | mIoU | Helped |
| Perturb I2T attention at the semantic grounding expert layer (ablation) | SD3 image generation · Image generation | image fidelity | Hurt |
| Guide denoising with a negatively perturbed I2T-attention sample (ablation) | SD3 image generation · Image generation | image quality | Helped |
Surprising findings
- Only a single MM-DiT block (layer 9) carries most of the semantic grounding; perturbing it degrades semantics far more than other layers.
- Padding-token attention maps in unconditional generation form coherent regions usable as unsupervised segmentation proposals.
Limitations
- Segmentation is evaluated without postprocessing or upsampling, which limits performance on very small objects.
- Diffusion models may ground class names differently from ground-truth annotations, causing a representation-annotation mismatch.
- Loss is applied to only one sweet-spot layer per backbone; broader layer and timestep exploration is left open.
Experimental setup
Datasets: SA-1B COCO DrawBench Pick-a-Pic PascalVOC COCO-Object Pascal Context-59 ADE20K COCO-Stuff-27 Cityscapes
Benchmarks: PascalVOC COCO-Object Pascal Context-59 ADE20K Cityscapes COCO-Stuff-27 T2I-CompBench++ Pick-a-Pic MS-COCO SA-1B captions
Metrics: mIoU pACC mACC CLIPScore T2I-CompBench++ attribute, relation, numeracy and complexity CLIP-I CLIP-T DINO
Baselines: DiffSegmenter iSeg ProxyCLIP CorrCLIP ReCO MaskCLIP MaskCut DiffSeg DiffCut
Builds on: Prompt-to-Prompt Stable Diffusion 3 SEG (smoothed energy guidance / attention perturbation) DiffSeg DiffCut rectified flow
Closest papers by meaning
- Text4Seg: Reimagining Image Segmentation as Text Generation (ICLR 2025 · similarity 0.90)
- EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models (ICLR 2024 · similarity 0.90)
- Rare Text Semantics Were Always There in Your Diffusion Transformer (NeurIPS 2025 · similarity 0.90)
- Unleashing the Potential of the Diffusion Model in Few-shot Semantic Segmentation (NeurIPS 2024 · similarity 0.89)
- SGDiff: Scene Graph Guided Diffusion Model for Image Collaborative SegCaptioning (AAAI 2025 · similarity 0.89)
- Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.