Research map / Vision-language and multimodal / Multimodal LLMs
EvolveDirector: Approaching Advanced Text-to-Image Generation with Large Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EvolveDirector, which trains a DiT text-to-image base model on images generated via public APIs of advanced T2I models, with a VLM (LLaVA-Next) dynamically discriminating, expanding, deleting and mutating training prompts to reach comparable quality with far less data (Edgen).
Problem
Training a text-to-image model comparable to advanced proprietary models using only publicly accessible resources, since many advanced models keep parameters private and offer only fee-based APIs; naive distillation from generated data requires ~10M samples, which is costly.
Approach
Online training loop in which a VLM acts as director: it pairwise-compares base-model and advanced-model images for each prompt, expands prompts where the base lags, deletes prompts where the base matches, and mutates to new prompts; when several advanced models are used, the VLM picks the best image among them as training target.
Key results
- Ablation: the full EvolveDirector with a 100K data cap reaches FID 7.45 and 48.53% human preference vs PixArt-α, comparable to the 10M-sample direct-distillation model (FID 7.36, 48.89%).
- Edgen, trained by selecting the best among multiple advanced models, outperforms DeepFloyd IF, Playground 2.5, Stable Diffusion 3, and Ideogram in human evaluation over 300 prompts.
- Ideogram images had the highest VLM select ratio overall (29.52%), while Stable Diffusion 3 led on text rendering (35.45%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train directly on 1M rather than 11M PixArt-α-generated samples | DiT pretrained on ImageNet and SAM · PixArt-α-generated data | FID: 7.36 -> 11.49 | Hurt |
| Train directly on 100K rather than 11M PixArt-α-generated samples | DiT pretrained on ImageNet and SAM · PixArt-α-generated data | FID: 7.36 -> 15.19 | Hurt |
| Add VLM discrimination-based sample deletion (ablation) | EvolveDirector on PixArt-α, 100K data cap · PixArt-α | FID: 15.19 -> 15.41 | No clear effect |
| Add prompt expansion (ablation) | EvolveDirector on PixArt-α, 100K data cap · PixArt-α | Human Evaluation: 32.22 -> 36.44 | Helped |
| Add VLM evaluation to guide reasoned prompt expansion and sample deletion (ablation) | EvolveDirector on PixArt-α, 100K data cap · PixArt-α | FID: 13.05 -> 7.61 | Helped |
| Add mutation of new prompts (ablation) | EvolveDirector on PixArt-α, 100K data cap · PixArt-α | FID: 7.61 -> 7.45 | Helped |
| Use EvolveDirector's dynamic curation instead of direct training on 100K generated samples | DiT pretrained on ImageNet and SAM · PixArt-α | FID: 15.19 -> 7.45 | Helped |
| Use EvolveDirector's dynamic curation instead of direct training on 100K generated samples | DiT pretrained on ImageNet and SAM · PixArt-α | Human Evaluation: 32.22 -> 48.53 | Helped |
| Use LLaVA-Next rather than other evaluated VLMs | EvolveDirector's VLM director · Human preference alignment | Discrimination: 0.840 | Mixed |
| Select training images from multiple advanced models using the VLM | Edgen · Human evaluation across tested prompts | Human preference | Helped |
| Use Stable Diffusion 3 rather than the other advanced models as the image source | VLM sample selection · Text generation | Select ratio: 35.45% | Helped |
Surprising findings
- Playground 2.5 was selected only 9.09% of the time on text-rendering prompts, the lowest of the four advanced models, suggesting its training data lacked text-in-image examples.
- Training from scratch on generated data with 100K samples without VLM curation gave a much weaker model (FID 15.19), indicating data volume alone is insufficient.
Limitations
- Model may inherit biases from the advanced models' generated images.
- Reliance on VLM evaluation may introduce the VLM's own biases into selection.
- Diminishing returns: a slight gap to the advanced model remains when approaching a single model.
Experimental setup
Datasets: ImageNet SAM LAION JourneyDB
Benchmarks: SAM captioned prompts (10K held-out prompts)
Metrics: FID human preference win rate select ratio
Baselines: PixArt-α DeepFloyd IF Playground 2.5 Stable Diffusion 3 Ideogram Midjourney
Builds on: PixArt-α DiT LLaVA-Next Gecko
Closest papers by meaning
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.87)
- OmniGen: Unified Image Generation (CVPR 2025 · similarity 0.87)
- Boosting Alignment for Post-Unlearning Text-to-Image Generative Models (NeurIPS 2024 · similarity 0.87)
- DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model (CVPR 2024 · similarity 0.87)
- JetFormer: An autoregressive generative model of raw images and text (ICLR 2025 · similarity 0.87)
- Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language Models (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.