Research map / Vision-language and multimodal / Human and embodied modeling
Instruct-Imagen: Image Generation with Multi-modal Instruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Instruct-Imagen unifies heterogeneous image generation tasks by expressing them as multi-modal instructions (natural-language text with referenced image contexts) and trains a cascaded Imagen-based diffusion model via retrieval-augmented training followed by multi-task instruction tuning.
Problem
Existing image generation models each accept specific conditioning modalities (text, subject, style, edge, mask) with ad hoc designs, and cannot follow complex multi-modal instructions or generalize to unseen task combinations.
Approach
Represent all image generation conditions as a uniform multi-modal instruction: a language instruction with reference markers linked to (marker + text, image) context pairs, encoded by a shared pre-trained diffusion backbone with one added cross-attention layer, trained first with retrieval-augmented (neighbor image-text) data and then on 11 instruction-formatted tasks.
Key results
- In-domain human study: Instruct-Imagen matches or surpasses prior task-specific models on most tasks (e.g., Txt2Img, Sty Gen, Sub Gen, Face Gen), per Figure 4.
- Masked image editing after fine-tuning on MagicBrush: human score 0.72 vs 0.43 for SDXL-inpainting and accuracy 0.57 vs 0.25.
- Retrieval-augmented training raises average human score from 0.55 to 0.79 in-domain and 0.53 to 0.59 zero-shot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Instruct-Imagen on MagicBrush | Instruct-Imagen · ImagenHub masked image editing | overall score: 0.37 -> 0.72 (+0.35) | Helped |
| Fine-tune Instruct-Imagen on MagicBrush | Instruct-Imagen · ImagenHub masked image editing | accuracy: 0.10 -> 0.57 (+0.47) | Helped |
| Add retrieval-augmented training (ablation) | Instruct-Imagen · In-domain evaluation | average overall score: 0.55 -> 0.79 (+0.25) | Helped |
| Add retrieval-augmented training (ablation) | Instruct-Imagen · Zero-shot evaluation | average overall score: 0.53 -> 0.59 (+0.06) | Helped |
Surprising findings
- Fine-tuned Instruct-Imagen produces artifacts in edited images because the cascaded super-resolution stage hallucinates details rather than copying pixels from the input.
- Multi-task baseline without instruction format fails to differentiate style from subject and gets 0 score on style transfer.
Limitations
- Cannot handle zero-shot image editing or inpainting due to lack of pixel consistency from cascaded low-resolution model.
- Fails to satisfy all conditions when instructions contain three or more conditions.
- Evaluation is mainly human study; evaluation suite promised but release not confirmed in text.
Experimental setup
Datasets: WebLI Internal Data WikiArt SuTI dataset Celeb-A Celeb-HQ Sketch2Image MagicBrush ImagenHub DreamBench CustomConcept101 StyleDrop
Benchmarks: ImagenHub DreamBench StyleDrop style images CelebA CelebA-HQ WebLI control evaluation CustomConcept101 MagicBrush
Metrics: human score O (SC x PQ) semantic consistency perceptual quality accuracy (SC=1 rate)
Baselines: SDXL ControlNet StyleDrop SuTI TamingEncoder StyleDrop+DreamBooth CustomDiffusion KOSMOS-G Ghiasi et al. style transfer SDXL-inpainting
Builds on: Imagen re-imagen ControlNet StyleDrop SuTI
Closest papers by meaning
- InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists (ICLR 2024 · similarity 0.91)
- OmniGen2: Towards Instruction-Aligned Multimodal Generation (CVPR 2026 · similarity 0.89)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.89)
- Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models (CVPR 2026 · similarity 0.88)
- A High Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation (ICLR 2026 · similarity 0.88)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.