Research map / Image generation / Text-to-image diffusion
The Crystal Ball Hypothesis in diffusion models: Anticipating object positions from initial noise
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies universal "trigger patches" in initial Gaussian noise that induce object generation at fixed locations in text-to-image diffusion, trains a noise-space detector to find them, and uses reject sampling on noise to improve positional diversity and prompt-position adherence.
Problem
Text-to-image diffusion models often fail to follow positional prompts and show positional bias, and it is unclear how the initial noise determines where objects appear in generated images.
Approach
Defines trigger patches in the initial noise via a posterior "trigger entropy" (variance of detected object box centers across prompts from one seed), shows they are universal across positions/prompts via injection, trains a detector operating directly on Gaussian noise to anticipate object positions, explains them as statistical outliers via energy-distance two-sample tests, and uses reject sampl
Key results
- Trigger-patch detector trained on Restricted set reaches mAP50 of 0.325, beating a permuted-annotation baseline by 0.124.
- Reject sampling with trigger-patch detection raises guidance success rate for left/right prompts from 57.08% (control) to 83.64%.
- Natural trigger patches reach 44.5% injection success rate vs 1.0% random and 8.5% resampling; diversity entropy 171.84 vs 135.97 for control.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train the detector on annotations from all prompts rather than sports-ball-only annotations (ablation) | Trigger-patch detector · Trigger-patch detector validation set | mAP50: 0.325 -> 0.333 | Helped |
| Add trigger-patch class labels to detector outputs (ablation) | Trigger-patch detector · Trigger-patch detector validation set | mAP50: 0.333 -> 0.091 | Hurt |
| Use randomized annotations instead of sports-ball annotations (ablation) | Trigger-patch detector · Trigger-patch detector validation set | mAP50: 0.325 -> 0.201 | Hurt |
| Reject-sample noise to remove detected trigger patches | Diffusion image generation · Location diversity | entropy: 135.97 -> 171.84 | Helped |
| Reject-sample noise until detected trigger patches align with positional prompts | Diffusion image generation · Positional prompt following | GSR (%): 57.08 -> 83.64 | Helped |
| Inject a natural low-trigger-entropy patch instead of a random patch | Trigger-patch injection · Trigger-patch injection | ISR (%): 1.0 -> 44.5 | Helped |
| Increase shifted-Gaussian standard deviation from 0.8 to 1.5 (ablation) | Trigger-patch injection · Trigger-patch injection | ISR (%): 8.5 -> 90.0 | Helped |
| Use sine-function patches with θ 0.15 instead of θ 0.08 (ablation) | Trigger-patch injection · Trigger-patch injection | ISR (%): 33.5 -> 81.0 | Helped |
| Use sine-function patches with θ 0.15 (ablation) | Trigger-patch injection · Generated image quality | image distortion | Hurt |
| Use a strong trigger patch aligned with the prompt rather than a contradicted trigger patch (ablation) | Diffusion image generation · Prompt: object on the right | Aligned (%): 35.0 -> 63.5 | Helped |
Surprising findings
- Trigger patches are class-agnostic: a detector trained with class labels performs much worse (mAP50 0.091), suggesting patches determine where, not what, is generated.
- When prompt and trigger patch conflict, the trigger patch wins in 32.5% of cases versus 35.0% following the prompt.
Limitations
- Multiple trigger patches within a single noise are not thoroughly explored; the entropy metric may fail in that case.
- Limited to five COCO classes and 25 prompts due to computational constraints.
- Manipulating trigger patches alone may not reach state-of-the-art prompt-following results.
Experimental setup
Datasets: COCO (five classes for prompts and annotations) self-generated noise-image dataset (20,000 noises)
Benchmarks: COCO validation (reference mAP50 for Faster R-CNN) custom noise-trigger dataset (Restricted, Augmented, Class-Specific, Permuted)
Metrics: mAP50 injection success rate (ISR) trigger entropy average box-center entropy (diversity) guidance success rate (GSR) energy distance p-value
Baselines: InitNO Resampling Random Control (standard generation)
Builds on: DDPM DDIM Generalized Focal Loss InitNO
Closest papers by meaning
- CCS: Controllable and Constrained Sampling with Diffusion Models via Initial Noise Perturbation (NeurIPS 2025 · similarity 0.89)
- Tracing the Roots: Leveraging Temporal Dynamics in Diffusion Trajectories for Origin Attribution (NeurIPS 2025 · similarity 0.89)
- Adjusting Initial Noise to Mitigate Memorization in Text-to-Image Diffusion Models (NeurIPS 2025 · similarity 0.88)
- InitNO: Boosting Text-to-Image Diffusion Models via Initial Noise Optimization (CVPR 2024 · similarity 0.88)
- Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models (CVPR 2024 · similarity 0.88)
- Theoretical insights for diffusion guidance: A case study for Gaussian mixture models (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.