Research map / Image generation / Text-to-image diffusion
Diffusion in Diffusion: Cyclic One-Way Diffusion for Text-Vision-Conditioned Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cyclic One-Way Diffusion (COW), a training-free method that injects visual-condition latents into a frozen Stable Diffusion model in a cyclic, one-way manner to preserve pixel-level conditioning while following text prompts.
Problem
Text-vision-conditioned generation (customization such as personalized generation, inpainting, style transfer, attribute editing) where low-level pixel information from a user-provided image must be preserved, while diffusion models' mutual interference among regions makes this hard; existing method
Approach
Treat diffusion as a controllable information flow: inverse the visual condition to latents, paste it on a gray background as seed initialization, and repeatedly replace latents of the condition region along a middle-stage denoising segment (disturb and reconstruct cycles), then replace again at a late step to preserve identity, all with a frozen pre-trained model.
Key results
- COW reaches 0.901 ID-Distance and 100% face detection rate, versus 0.408 for SD inpainting and 1.361 for DreamBooth, with 6 s generation time versus 732 s for DreamBooth.
- In a human study with 50 participants (3,000 responses over 600 groups), COW gets 69.73% condition consistency and 51.87% general fidelity, the highest among five methods.
- COW's CLIP-T is 0.306, second best behind DreamBooth's 0.329.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COW rather than DreamBooth | TV2I generation · TV2I | Condition Consistency: 09.60% -> 69.73% | Helped |
| Use COW rather than DreamBooth | TV2I generation · TV2I | General Fidelity: 28.73% -> 51.87% | Helped |
| Use COW rather than DreamBooth | TV2I generation · TV2I | CLIP-T: 0.329 -> 0.306 | Hurt |
| Use COW rather than SD inpainting | TV2I generation · TV2I | ID-Distance: 0.408 -> 0.901 | Hurt |
| Use gray rather than white, black, or random-noise seed background (ablation) | COW seed initialization · TV2I | image quality | Helped |
| Add seed initialization rather than omit it (ablation) | COW · TV2I | visual-condition fidelity | Helped |
| Use cycle position 700⇒500 rather than 900⇒700 (ablation) | COW · TV2I | Human Rating: 1.5 -> 4.6 | Helped |
| Use cycle position 700⇒500 rather than 800⇒400 (ablation) | COW · TV2I | Human Rating: 3.2 -> 4.6 | Helped |
| Use cycle position 700⇒500 rather than 500⇒300 (ablation) | COW · TV2I | Human Rating: 0.8 -> 4.6 | Helped |
| Reduce cycle count from 10 to 5 (ablation) | COW at position 700⇒500 · TV2I | Human Rating: 4.6 -> 2.5 | Hurt |
| Increase cycle count from 10 to 20 (ablation) | COW at position 700⇒500 · TV2I | Time Cost: 5.21 -> 8.83 | Hurt |
Surprising findings
- SD inpainting, which exactly preserves pixels, attains the best ID-Distance among baselines but scores very low on human condition consistency (6.33%).
Limitations
- Struggles with extremely strong conflicts between visual and text conditions (e.g., profile text with a front-face condition).
- Fails on orientation mismatch between the visual condition and the text prompt, and on face-covering text instructions with unobstructed faces.
- Evaluation limited to face conditions from CelebAMask-HQ with a small set of 200 images and 300 texts.
Experimental setup
Datasets: CelebAMask-HQ
Benchmarks: CelebAMask-HQ face masks with text prompts (200 images, 300 texts)
Metrics: CLIP-T ID-Distance Face Detection Rate Time Condition Consistency (human preference) General Fidelity (human preference)
Baselines: DreamBooth Textual Inversion ControlNet SD inpainting
Builds on: DDPM DDIM Stable Diffusion DreamBooth Textual Inversion
Closest papers by meaning
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.90)
- One-step Diffusion with Distribution Matching Distillation (CVPR 2024 · similarity 0.89)
- AdaDiff: Adaptive Step Selection for Fast Diffusion Models (AAAI 2025 · similarity 0.89)
- Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency (ICML 2025 · similarity 0.89)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.89)
- Smooth Diffusion: Crafting Smooth Latent Spaces in Diffusion Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.