Research map / Image generation / Text-to-image diffusion
Learning Visual Generative Priors without Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Lumos, a pure vision image-to-image latent diffusion prior trained self-supervised on ~190M in-the-wild images conditioned on DINO features, then transferred to T2I, novel view synthesis, and image-to-video with less text data.
Problem
Text-to-image generative priors depend heavily on high-quality text-image pairs, which are expensive to scale; the paper asks whether a visual generative prior can be learned without text.
Approach
Learn an image-to-image diffusion model conditioned on frozen self-supervised vision encoder features (e.g., DINO) from unlabeled images, use it as an upstream visual prior, and fine-tune it to text-to-image, novel view synthesis, and image-to-video tasks, arguing texture modeling rather than cross-modal alignment is what a generative prior needs.
Key results
- Lumos-T2I reaches COCO FID-30K 12.20 (6.44 with long captions) using 30M text-image pairs and 65k steps, competitive with larger T2I models.
- Lumos-NVS achieves PSNR 19.63 at 0° elevation and 20.28 at 30°, outperforming Zero123-XL (19.20) and SyncDreamer (18.98) at 30°.
- On MSR-VTT zero-shot I2V, the I2I prior yields FVD 271.00 versus 313.50 for the T2I prior and 1490.64 without prior.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Lumos-I2I rather than RCG as the image-to-image generative prior | I2I generation · MS-COCO and ImageNet-1K | FID: 12.70 -> 4.82 (MS-COCO); 4.89 -> 2.60 (ImageNet-1K) | Helped |
| Use an I2I prior rather than no prior | Text-to-image generation · MS-COCO | CLIP Score 30k: Figure 2a reports 0.5 less degradation in CLIP score compared with w/o I2I prior. | Helped |
| Scale I2I pre-training data from 10M to 200M images (ablation) | Lumos-I2I transferred to T2I · MS-COCO | FID | Helped |
| Use local patch features instead of a global [CLS] token (ablation) | Lumos-I2I · I2I generation | FID | Helped |
| Use local patch features instead of a global [CLS] token (ablation) | Lumos-I2I transferred to T2I · Text-to-image generation | FID | Hurt |
| Use a self-supervised vision encoder (DINO or MoCoV3) instead of CLIP (ablation) | Lumos-I2I · I2I generation | FID | Helped |
| Use DINO instead of CLIP as the I2I vision encoder (ablation) | Lumos-I2I transferred to T2I · Text-to-image generation | FID | Helped |
| Use T5 instead of CLIP as the text encoder (ablation) | T2I with a unimodal vision encoder such as DINO · Text-to-image generation | FID | Helped |
| Use an I2I prior rather than a T2I prior (ablation) | Novel view synthesis · GSO | PSNR | Helped |
| Use an I2I prior rather than a T2I prior (ablation) | Image-to-video generation · UCF-101 | FVD: 532.77 -> 399.59 | Helped |
| Use an I2I prior rather than a T2I prior (ablation) | Image-to-video generation · MSR-VTT | KVD: 19.61 -> 21.16 | Hurt |
| Use an I2I prior rather than no prior (ablation) | Image-to-video generation · UCF-101 | FVD: 2892.09 -> 399.59 | Helped |
Surprising findings
- DINO, a self-supervised encoder, converges slower early on than CLIP for T2I but achieves better final FID.
- Fine-grained local condition features speed up I2I training but make the pre-trained model more dependent on conditions, hindering transfer to downstream tasks.
Limitations
- T2I fine-tuning still uses 30M text-image pairs, so text data is not eliminated entirely.
- Downstream gains depend on the choice of vision encoder and condition features, with trade-offs between convergence speed and transferability.
- Compute and training hardware are not reported in the main text.
Experimental setup
Datasets: LAION-5B COYO-700M SAM JourneyDB ImageNet-1K Objaverse WebVid-10M OpenVid-1M MS-COCO GenEval DPG-Bench GSO
Benchmarks: MS-COCO FID-30K ImageNet-1K FID GenEval DPG-Bench GSO novel view synthesis UCF-101 I2V MSR-VTT I2V
Metrics: FID CLIP score GenEval overall DPG-Bench average PSNR SSIM LPIPS FVD KVD PIC
Baselines: PixArt-α SDv1.5 SDXL FLUX Zero123 Zero123-XL SyncDreamer RCG Stable Video Diffusion
Builds on: latent diffusion models DiT PixArt-α Zero-1-to-3 RCG
Closest papers by meaning
- Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency (ICLR 2024 · similarity 0.89)
- ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations (CVPR 2024 · similarity 0.89)
- MARS: Mixture of Auto-Regressive Models for Fine-grained Text-to-image Synthesis (AAAI 2025 · similarity 0.88)
- Localizing and Editing Knowledge In Text-to-Image Generative Models (ICLR 2024 · similarity 0.88)
- Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models (CVPR 2026 · similarity 0.87)
- Compositional Scene Understanding through Inverse Generative Modeling (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.