Research map / Image generation / Text-to-image diffusion
ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ECLIPSE, a non-diffusion text-to-image prior for unCLIP models trained with a projection MSE loss plus a CLIP-style contrastive loss, reaching competitive compositional performance with a 33M-parameter prior and few image-text pairs.
Problem
Text-to-image unCLIP priors add about a billion parameters and require large high-quality image-text datasets, making them resource-intensive; it is unclear whether the prior or scale drives compositional performance.
Approach
Replace the diffusion prior with a small non-diffusion transformer prior trained with an MSE projection loss plus a CLIP contrastive loss that distills the semantic text-image alignment of pre-trained vision-language models into the prior, improving generalization of compositions with little data.
Key results
- ECLIPSE 33M-parameter prior with Kandinsky v2.2 decoder trained on 5M LAION-HighRes pairs reaches 63.36% average PickScore preference against big SOTA models.
- ECLIPSE priors achieve an average of 71.6% PickScore preference over baseline priors in resource-limited settings.
- ECLIPSE prior with Karlo decoder on MSCOCO gets color score 0.5965 on T2I-CompBench, above Karlo's 0.5127.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CLIP contrastive loss to the projection prior (ablation) | Kandinsky v2.2 prior trained on HighRes · T2I-CompBench Color | color: 0.5379 -> 0.6119 | Helped |
| Add CLIP contrastive loss to the projection prior (ablation) | Kandinsky v2.2 prior trained on HighRes · T2I-CompBench Shape | shape: 0.4983 -> 0.5429 | Helped |
| Add CLIP contrastive loss to the projection prior (ablation) | Kandinsky v2.2 prior trained on HighRes · T2I-CompBench Texture | texture: 0.5217 -> 0.6165 | Helped |
| Add CLIP contrastive loss to the projection prior (ablation) | Kandinsky v2.2 prior trained on HighRes · T2I-CompBench Spatial | spatial: 0.1573 -> 0.1903 | Helped |
| Add CLIP contrastive loss to the projection prior (ablation) | Kandinsky v2.2 prior trained on MSCOCO · T2I-CompBench Color | color: 0.4678 -> 0.5785 | Helped |
| Add CLIP contrastive loss to the projection prior (ablation) | Karlo prior trained on MSCOCO · T2I-CompBench Shape | shape: 0.4421 -> 0.5063 | Helped |
| Add CLIP contrastive loss to the projection prior (ablation) | Karlo prior trained on CC3M · T2I-CompBench Texture | texture: 0.4948 -> 0.5881 | Helped |
| Add CLIP contrastive loss to the projection prior (ablation) | Karlo prior trained on CC12M · T2I-CompBench Spatial | spatial: 0.1318 -> 0.1625 | Helped |
| Replace the projection prior with a diffusion prior (ablation) | Kandinsky v2.2 prior trained on HighRes · T2I-CompBench Texture | texture: 0.6165 -> 0.5067 | Hurt |
| Replace the projection prior with a diffusion prior (ablation) | Kandinsky v2.2 prior trained on MSCOCO · T2I-CompBench Color | color: 0.5785 -> 0.4646 | Hurt |
| Replace the projection prior with a diffusion prior (ablation) | Karlo prior trained on MSCOCO · T2I-CompBench Spatial | spatial: 0.1574 -> 0.1240 | Hurt |
| Replace the projection prior with a diffusion prior (ablation) | Karlo prior trained on CC3M · T2I-CompBench Shape | shape: 0.5091 -> 0.4809 | Hurt |
Surprising findings
- Adding more diffusion prior steps and added sampling noise in the diffusion prior degrade human-preference performance.
- The original 1B-parameter Kandinsky v2.2 diffusion prior is preferred over the 33M ECLIPSE prior, which the authors attribute to aesthetic data quality.
Limitations
- Image quality (zero-shot FID and aesthetics) lags the original large priors and depends heavily on training dataset quality.
- Evaluation of preferences relies on PickScore as a proxy for human preference rather than large-scale human studies.
- Non-diffusion prior forgoes classifier-free guidance, which the authors note may reduce compositional capability.
Experimental setup
Datasets: MSCOCO CC3M CC12M LAION-HighResolution
Benchmarks: T2I-CompBench MS-COCO 30k zero-shot FID
Metrics: zero-shot FID T2I-CompBench color/shape/texture/spatial/non-spatial scores PickScore preference
Baselines: Stable Diffusion v1.4 Stable Diffusion v2.1 Würstchen Kandinsky v2.1 Kandinsky v2.2 DALL-E-2 Karlo Projection prior Diffusion-Baseline prior
Builds on: unCLIP / DALL-E-2 CLIP contrastive loss Karlo Kandinsky
Closest papers by meaning
- Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.90)
- Learning Visual Generative Priors without Text (CVPR 2025 · similarity 0.89)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.88)
- ORCA: Hunting Compositional Failures in Text-to-Image Diffusion (NeurIPS 2026 · similarity 0.88)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.88)
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.