Research map / Image generation / Text-to-image diffusion
InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes INITNO, a training-free method that optimizes the initial Gaussian noise of Stable Diffusion, using cross-attention response and self-attention conflict scores plus a KL distribution-alignment loss, to produce images that better match text prompts.
Problem
Text-to-image diffusion models often fail to faithfully render all prompt subjects and attribute bindings (subject neglect, subject mixing, incorrect binding), which the paper attributes to invalid initial noise.
Approach
Partition the initial latent space into valid and invalid noise using attention-based scores (cross-attention response and self-attention conflict), and optimize a Gaussian's mean and std of the initial noise with a joint loss including a KL term that keeps the noise near N(0,1), rather than updating the noisy image at each denoising step as Attend-and-Excite does.
Key results
- In the user study with 12 volunteers, INITNO was preferred in 63.33% of selections, versus 14.17% for Attend-and-Excite and 4.17% for Stable Diffusion.
- Inference takes 18.93 seconds per image versus 8.34 seconds for SD on a single Tesla V100, about 2.3x slower.
- Achieves higher CLIP image-text and text-text similarity than Attend-and-Excite, A-STAR, Divide-and-Bind and other baselines across the three prompt sets (per Fig. 7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use INITNO initial-noise optimization instead of the evaluated baselines | Stable Diffusion v1.4 · Animal-Animal, Animal-Object, and Object-Object datasets | CLIP image-text similarity and CLIP text-text similarity | Helped |
| Use INITNO instead of the evaluated baselines | Stable Diffusion v1.4 · User study | vote share: 63.33% | Helped |
| Add self-attention conflict loss (ablation) | INITNO · Apple-and-pear prompt | subject mixing | Helped |
| Replace the self-attention map with cross-attention maps in the conflict loss (ablation) | INITNO · Text-to-image generation | generation quality | Hurt |
| Remove distribution alignment loss (ablation) | INITNO · Text-to-image generation | image quality | Hurt |
| Add cross-attention response loss (ablation) | Attend-and-Excite · Apple-and-pear prompt | subject alignment | Mixed |
| Add self-attention conflict loss (ablation) | Attend-and-Excite · Apple-and-pear prompt | subject mixing | Helped |
Surprising findings
- A cross-attention conflict loss (from A-STAR) did not produce satisfactory results, which the authors attribute to inaccurate CLIP text embeddings, while self-attention maps worked better.
Limitations
- About 2.3x slower inference than vanilla Stable Diffusion due to noise optimization.
- Some intricate prompts cannot find valid noise within the optimization budget and fall back to the best noise in a pool.
- Evaluation relies on CLIP similarity and a small user study on three prompt categories.
Experimental setup
Datasets: Animal-Animal Animal-Object Object-Object
Benchmarks: Animal-Animal Animal-Object Object-Object
Metrics: CLIP Image-Text Similarity (full prompt) CLIP Image-Text Similarity (minimum object) CLIP Text-Text Similarity user study preference
Baselines: Stable Diffusion Composable Diffusion Structure Diffusion Attend-and-Excite Divide-and-Bind A-STAR
Builds on: Stable Diffusion Attend-and-Excite BoxDiff
Closest papers by meaning
- Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.92)
- Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.90)
- Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Models (NeurIPS 2025 · similarity 0.90)
- InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation (ICLR 2024 · similarity 0.90)
- On Discrete Prompt Optimization for Diffusion Models (ICML 2024 · similarity 0.90)
- Rethinking Training for De-biasing Text-to-Image Generation: Unlocking the Potential of Stable Diffusion (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.