Research map / Image generation / Text-to-image diffusion
Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Noise Diffusion, a training-free method that optimizes the initial noisy latent of Stable Diffusion using a VQA score from a large vision-language model, updating via forward-diffusion steps selected by gradient information.
Problem
Text-to-image diffusion models often fail to semantically align generated images with prompts (missing objects, wrong attributes or relations); prior initial-latent optimization (InitNo) is limited to a local region near the starting point and relies only on attention maps.
Approach
Uses an LVLM's VQA score (P("Yes") for "Does this figure show 'prompt'?") as the supervision signal to optimize the initial noisy latent, updating it with a diffusion-forward-style step (preserving the Gaussian distribution), with a score-aware step size and gradient-based selection among sampled noises, plus a theoretical condition for score improvement.
Key results
- On simple prompts at epoch 50, Noise Diffusion reaches a VQA score of 0.9790, exceeding InitNo's average by 0.1073.
- On complex prompts at epoch 50, VQA score of 0.9578, surpassing InitNo's average by 0.1930.
- Average CLIP and VQA scores of refined images improve across SD V-1.4, V-1.5, V-2.0, and V-2.1 with all four LVLMs tested.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Noise Diffusion instead of InitNo | Stable Diffusion V-1.4 · Simple cases | VQA score: 0.9352; surpassing InitNo by 0.1073 at epoch 50 | Helped |
| Use Noise Diffusion instead of InitNo | Stable Diffusion V-1.4 · Complex cases | VQA score: surpassing InitNo by 0.1144 at epoch 10; surpassing InitNo by 0.1930 at epoch 50 | Helped |
| Use gradient-guided noise selection rather than random noise selection (ablation) | Noise Diffusion · a bear wearing a T-shirt | VQA score | Helped |
| Use a score-adjusted step size rather than a fixed step size (ablation) | Noise Diffusion · a bear wearing a T-shirt | semantic alignment | Helped |
| Use Noise Diffusion rather than Random Diffusion (ablation) | Noise Diffusion · a bear wearing a T-shirt | semantic alignment | Helped |
| Directly perturb the latent with PGD (ablation) | Stable Diffusion V-1.4 · a bear wearing a T-shirt | semantic alignment | No clear effect |
| Adjust the mean and variance using Adam (ablation) | Stable Diffusion V-1.4 · a bear wearing a T-shirt | image quality | Hurt |
| Randomly replace the current noise at each timestep (ablation) | Stable Diffusion V-1.4 · a bear wearing a T-shirt | semantic alignment | Hurt |
Surprising findings
- Mean-Variance (InitNo-style) perturbation degrades visual quality by epoch 50, suggesting the initial latent lies in a low-score region with no nearby high-score points.
Limitations
- Substantial added computation per image (about 6.71 s per optimization epoch).
- Relies on the LVLM's VQA score as the sole supervision signal; theoretical guarantee depends on bounded second derivatives assumption and gradient approximation that ignores dependence of noise predictions on zT.
- Evaluated only on prompts from a limited set of simple and complex object-composition cases.
Experimental setup
Datasets: simple prompts from Chefer et al. (GSN) complex prompts built by replacing conjunctions with spatial terms
Benchmarks: simple prompt set complex prompt set
Metrics: VQA score CLIP score
Baselines: InitNo Stable Diffusion PGD Mean-Variance Random Sampling Random Diffusion
Builds on: InitNo DDIM Generative Semantic Nursing VQAScore
Closest papers by meaning
- InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization (CVPR 2024 · similarity 0.92)
- Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.91)
- Progressive Text-to-Image Diffusion with Soft Latent Direction (AAAI 2024 · similarity 0.90)
- Rare Text Semantics Were Always There in Your Diffusion Transformer (NeurIPS 2025 · similarity 0.90)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.90)
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.