Research map / Image generation / Text-to-image diffusion
Text-Image Alignment for Diffusion-Based Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TADP, which uses BLIP-2 captions and caption modifiers (including Textual Inversion/DreamBooth) to align text prompts with images for diffusion-backbone perception, improving segmentation, depth, and cross-domain detection/segmentation.
Problem
How to prompt diffusion-pretrained backbones for discriminative vision tasks; unaligned text prompts (averaged class embeddings) degrade cross-attention features, and cross-domain transfer lacks target-domain alignment.
Approach
Replace averaged CLIP EOS class embeddings with automatically generated image-aligned BLIP-2 captions as the diffusion conditioning, and add target-domain caption modifiers (hand-crafted style text, Textual Inversion, DreamBooth) for cross-domain transfer.
Key results
- TADP improves Pascal VOC segmentation by about 4.0 mIoU and ADE20K by 1.7 mIoU over VPD, and reaches 0.225 RMSE on NYUv2 depth (new SOTA).
- Cross-domain Cityscapes to Dark Zurich-val reaches 42.8 mIoU with null captions (+5.8 over PTDiffSeg) and Textual Inversion reaches 60.8 mIoU on Nighttime Driving.
- Pascal VOC to Watercolor2k object detection reaches 72.1-72.2 AP50 across caption variants, setting SOTA for that setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use image-aligned BLIP captions instead of the VPD prompt, with latent scaling | TADP · Pascal VOC2012 segmentation | mIoU: +4.0 mIoU | Helped |
| Use image-aligned BLIP captions instead of the VPD prompt, with latent scaling | TADP · ADE20K segmentation | mIoU: +1.7 mIoU | Helped |
| Use image-aligned BLIP captions instead of the VPD prompt, with latent scaling | TADP · NYUv2 depth estimation | RMSE: 0.235 -> 0.225 | Helped |
| Add latent scaling to the encoded image (ablation) | VPD · Pascal VOC2012 segmentation | mIoUss: 82.34 -> 83.06 | Helped |
| Replace averaged EOS embeddings with class-word embeddings (ablation) | Pascal VOC2012 segmentation · Pascal VOC2012 | mIoU: -0.3 mIoU | Hurt |
| Use a generic space-separated string of class names instead of averaged EOS embeddings (ablation) | Pascal VOC2012 segmentation · Pascal VOC2012 | mIoU: +1.0 mIoU | Helped |
| Increase the minimum BLIP caption length from 0 to 40 tokens (ablation) | TADP · Pascal VOC2012 segmentation | mIoUss: 86.36 -> 87.11 | Helped |
| Use oracle captions listing classes present in the image instead of automatic captions (ablation) | TADP · ADE20K segmentation | mIoU | Helped |
| Add target-domain information using a learned Textual Inversion token | TADP · Nighttime Driving cross-domain segmentation | mIoU: +3.3 mIoU | Helped |
| Add target-domain information using a learned Textual Inversion token | TADP · Watercolor2k cross-domain detection | AP: +1.4 AP | Helped |
| Add target-domain information using a learned Textual Inversion token | TADP · Comic2k cross-domain detection | AP: +2.1 AP | Helped |
| Append an off-target domain style to the caption (ablation) | TADP cross-domain models · Cross-domain segmentation and object detection | performance | Hurt |
Surprising findings
- Averaged CLIP EOS class embeddings degrade cross-attention maps, and even a plain string of class names outperforms them.
- Diffusion pre-training with null captions alone already achieves SOTA on several cross-domain benchmarks.
Limitations
- Oracle captions (ground-truth class names) show large headroom, but a realistic closed-vocabulary captioner is not explored.
- Cross-domain gains from text alignment are dataset-dependent; misaligned target-domain text hurts performance.
- Requires target-domain unlabeled images for Textual Inversion/DreamBooth personalization.
Experimental setup
Datasets: Pascal VOC2012 ADE20K NYUv2 Cityscapes LAION-400M
Benchmarks: Pascal VOC2012 ADE20K NYUv2 Watercolor2k Comic2k Dark Zurich-val Nighttime Driving
Metrics: mIoU RMSE AP AP50
Baselines: VPD VPD(R) VPD(LS) PTDiffSeg DAFormer Refign-DAFormer DASS-Detector CLIP the gap ZoeDepth AiT SwinV2-L
Builds on: VPD Stable Diffusion BLIP-2 Textual Inversion DreamBooth
Closest papers by meaning
- Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.90)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.90)
- Diffusion Bridge: Leveraging Diffusion Model to Reduce the Modality Gap Between Text and Vision for Zero-Shot Image Captioning (CVPR 2025 · similarity 0.89)
- Improving Long-Text Alignment for Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.89)
- Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.89)
- De-Diffusion Makes Text a Strong Cross-Modal Interface (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.