Research map / Image generation / Text-to-image diffusion
OmniGen: Unified Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces OmniGen, a single ~3.8B diffusion transformer (Phi-3 initialized, SDXL VAE, rectified flow) trained on the X2I multi-task dataset to handle text-to-image, editing, subject-driven, and visual-conditional generation end-to-end without plugins.
Problem
Image generation lacks a unified model that handles many tasks within one framework via instructions; existing diffusion models need task-specific modules, extra encoders, and multi-step workflows.
Approach
A minimal architecture (VAE + pretrained LLM transformer) that jointly models interleaved text and image tokens with rectified flow, using causal attention across elements but bidirectional attention within each image, plus a weighted loss to prevent copy-shortcuts in editing, trained on a unified any-to-image dataset.
Key results
- On GenEval, OmniGen (3.8B) reaches 0.70 overall, comparable to SD3 (8.0B+4.7B text encoder, 0.68) and above SD-XL (0.55) and DALL-E 2 (0.52).
- On EMU-Edit, OmniGen obtains CLIP-T 0.231 and CLIP-I 0.829, comparable to the best editing baselines such as EMU-Edit (0.231/0.859).
- On DreamBench, OmniGen achieves CLIP-T 0.315 (best listed) with CLIP-I 0.801, versus Kosmos-G 0.287/0.847 and SuTI 0.304/0.819.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OmniGen rather than SD3 | Text-to-image generation · GenEval | Overall: 0.68 -> 0.70 | Helped |
| Use OmniGen rather than Null-Text | Image editing · EMU-Edit test | CLIP-T: 0.236 -> 0.231 | Hurt |
| Use OmniGen rather than EMU-Edit | Image editing · EMU-Edit test | CLIP-I: 0.859 -> 0.829 | Hurt |
| Use OmniGen rather than DreamBooth | Subject-driven generation · DreamBench | CLIP-T: 0.305 -> 0.315 | Helped |
| Use OmniGen rather than Kosmos-G | Subject-driven generation · DreamBench | CLIP-I: 0.847 -> 0.801 | Hurt |
| Use OmniGen rather than ControlNet++ | Controllable generation · Segmentation mask | mIoU: 43.64 -> 40.06 | Hurt |
| Use OmniGen rather than UniControl | Controllable generation · Canny edge | F1 Score: 30.82 -> 38.96 | Helped |
| Use OmniGen rather than ControlNet++ | Controllable generation · HED edge | SSIM: 0.8097 -> 0.8332 | Helped |
| Use OmniGen rather than ControlNet++ | Controllable generation · Depth map | RMSE: 28.32 -> 31.71 | Hurt |
| Use the modified bidirectional attention within each image sequence (ablation) | OmniGen · Image generation | visual quality | Helped |
| Add weighted loss for changed image regions (ablation) | OmniGen image editing · Image editing | edit quality | Helped |
| Use VAE rather than CLIP to encode input images (ablation) | OmniGen · EMU-Edit and DreamBench | CLIP-T and CLIP-I: 0.232 vs 0.231; 0.820 vs 0.829; 0.312 vs 0.315; 0.789 vs 0.801 | Mixed |
Surprising findings
- Without any explicit training on the task, OmniGen handles ControlNet-style reference-based pose/condition following in a single step by reading a reference image directly.
- Using a 2D causal attention mask borrowed from LLMs (unidirectional within images) produces noisy, distorted images, so bidirectional attention within each image is needed.
Limitations
- Limited text rendering capability.
- Outputs may contain undesired details such as abnormal hands.
- Previously unseen image types (e.g., surface normal maps) are not handled as expected.
Experimental setup
Datasets: X2I Recap-DataComp SAM-LLaVA ShareGPT4V LAION-Aesthetic ALLaVA-4V DOCCI DenseFusion JourneyDB MagicBrush InstructPix2Pix SEED-edit
Benchmarks: GenEval EMU-Edit DreamBench MultiGen controllable generation benchmark (Canny, HED, Depth, Segmentation) FSS (in-context unseen segmentation)
Metrics: GenEval overall score CLIP-T CLIP-I mIoU F1 score SSIM RMSE
Baselines: SD3 SD-XL DALL-E 3 IF-XL InstructPix2Pix MagicBrush Emu-Edit DreamBooth Kosmos-G ControlNet ControlNet++ UniControl
Builds on: rectified flow Phi-3 SDXL VAE ControlNet InstructPix2Pix
Closest papers by meaning
- OmniGen2: Towards Instruction-Aligned Multimodal Generation (CVPR 2026 · similarity 0.90)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.89)
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.89)
- ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer (ICLR 2025 · similarity 0.88)
- Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.88)
- Instruct-Imagen: Image Generation with Multi-modal Instruction (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.