Research map / Image generation / Text-to-image diffusion
Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free text-to-image method that inverts a frozen BLIP-2 VLM by optimizing VAE latents with augmentation regularization, combined with Stable Diffusion SDS for fidelity and EMA-restart.
Problem
Text-to-image diffusion models struggle to follow complex, multi-attribute compositional prompts, with weak controllability and text-image alignment.
Approach
Treats conditional generation as model inversion of a discriminative VLM: images are optimized (in a VAE latent space) to minimize BLIP-2's image-text matching and captioning losses, regularized by averaging over random augmentations (excluding flips and crops) and balanced with SDS gradients from Stable Diffusion, plus an EMA-restart schedule.
Key results
- On T2I-CompBench, the method reaches 0.8162 color, 0.6209 shape, 0.7202 texture, 0.1506 spatial, 0.3215 non-spatial and 0.3529 complex, near SOTA but below BLIP-2 Inv on color.
- Individual BLIP-2 inversion reaches 0.8639 color and 0.6686 shape, showing strong attribute binding but weak spatial scores (0.1008).
- Augmentation regularization with 30 augmentations raises BLIP-VQA color score from 0.6561 to 0.8639 over no augmentation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add augmentation regularization (30 augmentations) (ablation) | BLIP-2 inversion · T2I-CompBench Attribute Binding | BLIP-VQA score: 0.6561 -> 0.8639 (color); 0.5078 -> 0.6686 (shape); 0.5371 -> 0.7311 (texture) | Helped |
| Add SDS to BLIP-2 inversion | BLIP-2 inversion · T2I-CompBench | BLIP-VQA score: 0.8639 -> 0.8162 (color); 0.6686 -> 0.6209 (shape); 0.7311 -> 0.7202 (texture); 0.1008 -> 0.1506 (spatial); 0.3260 -> 0. | Mixed |
| Scale SDS weight from 800 to 400 with CFG 30 (ablation) | BLIP-2 + Stable Diffusion · T2I-CompBench Attribute Binding | VQA score: 63.34 -> 69.43 | Helped |
| Increase SDS weight schedule from 1000 → 500 to 1500 → 500 at CFG 30 (ablation) | BLIP-2 + Stable Diffusion · T2I-CompBench Attribute Binding | VQA score: 67.43 -> 63.34 | Hurt |
| Increase BLIP-2 weight from 1 to 2 at frequency 1 (ablation) | BLIP-2 + Stable Diffusion · T2I-CompBench Attribute Binding | VQA score: 59.97 -> 67.43 | Helped |
| Increase BLIP-2 weight from 2 to 3 at frequency 1 (ablation) | BLIP-2 + Stable Diffusion · T2I-CompBench Attribute Binding | VQA score: 67.43 -> 62.36 | Hurt |
| Use EMA restart with start 80 and interval 20 instead of no EMA (ablation) | BLIP-2 + Stable Diffusion · T2I-CompBench Attribute Binding | VQA score: 67.43 -> 71.77 | Helped |
| Add random latent noise with scale 0.2 instead of scale 0.3 (ablation) | BLIP-2 + Stable Diffusion · T2I-CompBench Attribute Binding | VQA score: 64.57 -> 71.04 | Helped |
| Add random horizontal flipping or random resized cropping (ablation) | BLIP-2 inversion augmentation · T2I-CompBench prompts | BLIP-2 loss | Hurt |
| Increase Stable Diffusion CFG from 7.5 to 30 | Stable Diffusion v1.5 · T2I-CompBench Attribute Binding | BLIP-VQA score: 0.3632 -> 0.4012 (color); 0.3490 -> 0.4018 (shape); 0.3985 -> 0.4386 (texture) | Helped |
Surprising findings
- Horizontal flip and random cropping hurt BLIP-2 inversion because they alter semantics such as left/right and object counts.
- BLIP-2 inversion alone is weak on spatial relations, and the method inherits this limitation.
Limitations
- Optimization-based generation is less stable than state-of-the-art diffusion models and sometimes yields mismatched or unrealistic images.
- Struggles with spatial relations because BLIP-2 is insensitive to location information.
- Only one referee (BLIP-2) is used; tested only on T2I-CompBench.
Experimental setup
Datasets: T2I-CompBench
Benchmarks: T2I-CompBench
Metrics: BLIP-VQA score attribute binding score (color, shape, texture) spatial score non-spatial score complex score
Baselines: Stable Diffusion v1.4 Composable Diffusion v2 Structured Diffusion v2 Attend-and-Excite v2 GORS DALLE-2 SDXL PixArt-alpha DALLE-3 SDS v1.5 BLIP-2 Inv
Builds on: score distillation sampling latent diffusion models BLIP-2 VQGAN-CLIP classifier-free guidance
Closest papers by meaning
- CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation (CVPR 2024 · similarity 0.89)
- A Simple Approach to Unifying Diffusion-based Conditional Generation (ICLR 2025 · similarity 0.89)
- DCI: Dual-Conditional Inversion for Boosting Diffusion-Based Image Editing (NeurIPS 2025 · similarity 0.88)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.88)
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.88)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.