Research map / Image generation / Text-to-image diffusion
Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Prompt-Free Diffusion, which replaces the CLIP text encoder of a latent diffusion T2I model with a visual encoder (SeeCoder) so generation is conditioned only on a reference image, optional structural conditioning, and noise.
Problem
Text prompts for personalized text-to-image generation are hard to engineer and often cannot capture fine visual details, while finetuning and ControlNet-style adapters have cost and generality limitations.
Approach
Replace the CLIP text encoder in a latent diffusion model with SeeCoder, a SWIN-L backbone plus multi-level decoder and query transformer that maps reference images into context embeddings for cross-attention, trained with the frozen diffuser so the encoder is reusable across T2I models without prompts.
Key results
- Prompt-Free Diffusion with SeeCoder-PA beats ControlNet Shuffle and Reference-Only in image-variation for both in-domain and out-of-domain samples (qualitative comparison, Figure 6).
- SeeCoder plugged into six other open-source T2I models without any retraining, demonstrating reusability (Figure 8).
- Prompt-Free Diffusion roughly matches the quality of prompt-based ControlNet+T2I with sophisticated prompts, between the top two prompt-complexity levels (qualitative, Figure 5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace text prompts with a reference image using Prompt-Free Diffusion | Prompt-Free Diffusion · T2I generation | image quality | Helped |
| Use Prompt-Free Diffusion instead of prompt-based ControlNet+T2I | Prompt-Free Diffusion · Prompt comparison | image quality | Mixed |
| Use SeeCoder-PA instead of SeeCoder (ablation) | Prompt-Free Diffusion · Image-Variation, no structural conditioning | image quality | Helped |
| Use Prompt-Free Diffusion instead of ControlNet Shuffle or Reference-Only | Image-Variation · In-domain samples | reference-image replication quality | Helped |
| Use Prompt-Free Diffusion instead of ControlNet Shuffle or Reference-Only | Image-Variation · Out-of-domain samples | reference-image replication quality | Helped |
| Use Prompt-Free Diffusion instead of Dual-ControlNet | Prompt-Free Diffusion · Structural-guided generation | texture, color, and style replication quality | Helped |
| Reuse the same pretrained SeeCoder with other T2I diffusers without training | Prompt-Free Diffusion · Seven open-sourced and in-house T2I models | adaptability | Helped |
| Use SeeCoder instead of ControlNet Shuffle or Reference-Only | Prompt-Free Diffusion · Image-Variation, out-of-domain samples | generalization quality | Mixed |
| Fine-tune SeeCoder on OAM-V2 and its synthesized images (ablation) | Prompt-Free Diffusion · Anime figure generation | generation quality | Helped |
| Use reference-image visual embeddings and pose conditioning for anime figure generation | Prompt-Free Diffusion · Anime figure generation | generation quality | Helped |
| Use SeeCoder visual embeddings with cloth-mask conditioning | Prompt-Free Diffusion · Virtual try-on | generation quality | Helped |
Surprising findings
- Versatile Diffusion, which finetunes the diffusion model for image variation, has the best overall performance in the image-variation comparison, outperforming the prompt-free approach on that test.
- The SeeCoder-PA variant with spatial embeddings performs better than SeeCoder when no structural conditioning is used.
Limitations
- Evaluation is largely qualitative, with no quantitative metrics reported in the main text.
- Relies on an in-house SD1.5-based diffuser for main experiments, which is not publicly available.
- Performance drops on out-of-domain reference images for ControlNet baselines and the method still has quality gaps on out-of-domain samples.
Experimental setup
Datasets: Laion2B-en COYO-700M
Baselines: Versatile Diffusion ControlNet Shuffle ControlNet Reference-Only Dual-ControlNet DreamBooth HR-VITON VITON-HD
Builds on: Stable Diffusion ControlNet CLIP Swin Transformer
Closest papers by meaning
- DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation (ICLR 2025 · similarity 0.93)
- Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.92)
- CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.91)
- Textualize Visual Prompt for Image Editing via Diffusion Bridge (AAAI 2025 · similarity 0.91)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.91)
- R&B: Region and Boundary Aware Zero-shot Grounded Text-to-image Generation (ICLR 2024 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.