Research map / Image generation / Text-to-image diffusion
DreamOmni: Unified Image Generation and Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DreamOmni, a unified DiT-based model (2.5B) that jointly trains text-to-image generation and multiple editing tasks, using a collage-based synthetic data pipeline built from stickers and geometric shapes.
Problem
Text-to-image foundation models are not designed to unify with downstream image editing tasks (instruction-based, inpainting, drag, reference-image generation), which typically need specialized plugins or input-channel changes, and high-quality editing data is hard to create and filter.
Approach
A single DiT-style framework that replaces the text encoder with a VLM (Qwen2-VL-7B) whose features are concatenated with noisy latents for joint self-attention, plus a synthetic collage data pipeline (stickers, shapes, masks, drag coordinates) that generates scalable editing and T2I training data, trained jointly with rectified flow.
Key results
- On GenEval, DreamOmni reaches an overall score of 0.70, matching SD3-Medium (0.70) and above SDXL (0.55) and SD 1.5 (0.43).
- On inpainting, DreamOmni achieves FID 0.8371 and LPIPS 0.1203, versus 1.3522/0.1560 for SD-inpainting.
- In framework comparison at 0.85B parameters, DreamOmni-V3 converges about four times faster than SD3-Medium.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a Unet-style long connection (ablation) | DreamOmni framework · T2I framework training | training convergence speed: four times faster | Helped |
| Concentrate DIT block computations on 2× downsampled latent (ablation) | DreamOmni-V2 · T2I framework | computational cost-effectiveness | Helped |
| Add synthetic data targeting quantity, color, and position accuracy | DreamOmni · GenEval | Overall | Helped |
| Jointly train T2I data with editing data | DreamOmni · Image editing | editing performance | Helped |
Surprising findings
- Concentrating DiT computation on the 2x downsampled latent was more cost-effective than higher-resolution attention, and Unet-style long residual connections markedly speed convergence.
Limitations
- Large-scale rotation drag edits can deform the edited object.
- Evaluation of editing relies largely on qualitative comparisons and self-constructed evaluation sets.
- Code and model are only promised for release ('will be released').
Experimental setup
Datasets: LAION synthetic collage data (stickers, shapes, masks, drag points) GenEval
Benchmarks: GenEval FID/LPIPS on inpainting and outpainting (own high-quality eval set) qualitative comparisons on instruction editing, drag editing, image-conditioned and subject-driven generation
Metrics: GenEval overall score FID LPIPS
Baselines: SDXL PixArt-alpha SD3-Medium SD 1.5 SD-Cascade ControlNet SD-Inpainting BLIP-Diffusion IP-Adapter MGIE InstructP2P
Builds on: Diffusion Transformer (DiT) Rectified Flow SDXL ControlNet
Closest papers by meaning
- ImgEdit: A Unified Image Editing Dataset and Benchmark (NeurIPS 2025 · similarity 0.88)
- FireEdit: Fine-grained Instruction-based Image Editing via Region-aware Vision Language Model (CVPR 2025 · similarity 0.86)
- OneIG-Bench: Omni-dimensional Nuanced Evaluation for Image Generation (NeurIPS 2025 · similarity 0.86)
- UltraEdit: Instruction-based Fine-Grained Image Editing at Scale (NeurIPS 2024 · similarity 0.86)
- ImagenHub: Standardizing the evaluation of conditional image generation models (ICLR 2024 · similarity 0.86)
- EditAR: Unified Conditional Generation with Autoregressive Models (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.