Research map / Image generation / Text-to-image diffusion
Open-Vocabulary Attention Maps with Token Optimization for Semantic Segmentation in Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OVAM, a training-free method that extracts diffusion cross-attention maps for any open-vocabulary word, plus a single-annotation token optimization that improves pseudo-mask quality for Stable Diffusion segmentation methods.
Problem
Diffusion-based pseudo-mask generation from cross-attention is limited to words present in the generation prompt, so classes not mentioned in the prompt cannot be segmented.
Approach
Use a separate attribution prompt's key projections with the pixel queries from the denoising UNet to form attention maps for arbitrary words, and optimize a single learned text-embedding token per class with BCE on one annotated image to improve attention masks, without retraining the diffusion model.
Key results
- Token optimization raises OVAM mIoU on COCO-cap from 58.2 to 69.2 and on VOC-sim from 70.4 to 82.5, outperforming DAAM and Attn2Mask baselines and DatasetDM.
- Grounded Diffusion with optimized tokens reaches 86.6 mIoU on VOC-sim, up from 62.1, the best reported result.
- Adding OVAM synthetic data to full VOC12 real data improves Mask2Former (Swin-B) ... up to 6.9% mIoU gains for some models per the paper's claim.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OVAM-optimized tokens instead of class-name tokens (ablation) | OVAM · VOC-sim | mIoU (%): 70.4 -> 82.5 | Helped |
| Use OVAM-optimized tokens instead of class-name tokens (ablation) | OVAM · COCO-cap | mIoU (%): 58.2 -> 69.2 | Helped |
| Use OVAM-optimized tokens | Grounded Diffusion · COCO-cap | mIoU (%): 50.2 -> 73.3 | Helped |
| Use OVAM-optimized tokens | DatasetDM · COCO-cap | mIoU (%): 59.3 -> 60.5 | Helped |
| Add OVAM synthetic training data to VOC (R: 11.5K) | Upernet (ResNet50) · VOC validation set | mIoU (%): 54.3 -> 58.1 | Helped |
| Add OVAM synthetic training data to VOC (R: 11.5K) | Mask2Former (Swin-B) · VOC validation set | mIoU (%): 82.8 -> 83.8 | Helped |
| Use self-attention refinement without dCRF (ablation) | OVAM · COCO-cap | mIoU (%): 58.2 -> 50.7 | Hurt |
| Use dCRF without self-attention refinement (ablation) | OVAM · COCO-cap | mIoU (%): 58.2 -> 50.7 | Hurt |
| Extract cross-attention only from 16x16 blocks (ablation) | OVAM · VOC-sim | mIoU (%): 70.4 -> 71.5 | Helped |
| Extract cross-attention only from 16x16 blocks (ablation) | OVAM + token optimization · COCO-cap | mIoU (%): 69.2 -> 68.2 | Hurt |
Surprising findings
- Optimized tokens improve Grounded Diffusion by a large margin (62.1 to 86.6 mIoU on VOC-sim) even though that method was trained with class names.
- Aggregating attention across all denoising timesteps is best, but with token optimization similar performance is reached using only the midpoint timestep.
Limitations
- Relies on manual annotation of evaluation masks by three annotators; evaluation limited to the 20 VOC classes.
- Requires at least one annotated image per class for token optimization, and tuned binarization hyperparameters (tau, alpha) differ between settings.
Experimental setup
Datasets: VOC12 COCO captions VOC-sim (generated) COCO-cap (generated)
Benchmarks: VOC-sim COCO-cap VOC validation set
Metrics: mIoU IoU
Baselines: DAAM Attn2Mask Grounded Diffusion DatasetDM
Builds on: DAAM Attn2Mask Grounded Diffusion DatasetDM Stable Diffusion
Closest papers by meaning
- Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.89)
- Training-Free Open-Vocabulary Segmentation with Offline Diffusion-Augmented Prototype Generation (CVPR 2024 · similarity 0.89)
- FreeGen: Bridging Visual-Linguistic Discrepancies Towards Diffusion-based Pixel-level Data Synthesis (AAAI 2025 · similarity 0.88)
- Text-Image Alignment for Diffusion-Based Perception (CVPR 2024 · similarity 0.88)
- SeeDiff: Off-the-Shelf Seeded Mask Generation from Diffusion Models (AAAI 2025 · similarity 0.88)
- Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.