Research map / Image generation / Text-to-image diffusion
Imagine Before You Draw: Visual Prompt Engineering for Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Visual Prompt Engineering (VPE), which has a unified autoregressive model first generate SigLIP 2 visual semantic tokens as a plan before image tokens, with a progressive schedule to handle train-inference gap, improving class-, text-, and editing-conditioned generation.
Problem
Direct text-to-image modeling is hard, and external two-stage pipelines (AR semantic tokens feeding a separate diffusion decoder) create an information bottleneck that hurts detail preservation in editing.
Approach
Insert SigLIP 2 visual tokens as autoregressively generated "visual prompts" before image tokens, integrated either internally (shared attention in a MoT model) or externally (AR + DiT), with progressive masking (discrete) or CE-loss coefficient scheduling (continuous) to bridge the train-inference gap and loss-scale imbalance.
Key results
- LlamaGen+VPE reaches FID 8.69 vs 24.56 for LlamaGen at epoch 100 on ImageNet 512x512 (no CFG).
- On PIE-Bench editing, Internal+VPE achieves PSNR 26.76 vs 19.92 for External+VPE at matched 4.57B params.
- External+VPE (4.16B) reaches GenEval overall 0.81, above 7B-scale baselines such as Janus-Pro (0.80) and MetaQuery-XL (0.80).
Surprising findings
- Internal+VPE gives much better editing preservation than External+VPE despite similar T2I quality and editing responsiveness (CLIP scores).
- Show-o2+VPE raises OCR accuracy on TextScenesHQ from 2.62% to 16.29% under matched data.
Limitations
- Validated only on relatively small models (up to about 4.6B); scaling behavior untested.
- Only SigLIP 2 studied as the semantic representation; CLIP and DINOv2 not explored.
- Progressive schedule hyperparameters not extensively tuned.
Experimental setup
Datasets: ImageNet-1K BLIP3o T2I data TextAtlas AnyWord-3M NHR-Edit UnicEdit ShareGPT4o Pico-Banana
Benchmarks: ImageNet 512x512 FID TextAtlas (TextScenesHQ, TextVisionBlend, StyleTextSynth) GenEval PIE-Bench
Metrics: FID IS Precision Recall CLIP score OCR accuracy F1 CER GenEval overall PSNR
Baselines: LlamaGen Transfusion Show-o2 SD3.5 Large Janus-Pro Emu3 MetaQuery-XL BLIP3o-Next X-Omni
Builds on: LlamaGen Transfusion Show-o2 BAGEL X-Omni
Closest papers by meaning
- Textualize Visual Prompt for Image Editing via Diffusion Bridge (AAAI 2025 · similarity 0.90)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.89)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- Visually Guided Decoding: Gradient-Free Hard Prompt Inversion with Language Models (ICLR 2025 · similarity 0.89)
- ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer (ICLR 2025 · similarity 0.89)
- PromptCoT: Align Prompt Distribution via Adapted Chain-of-Thought (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.