Research map / Vision-language and multimodal / Multimodal LLMs

OmniGen2: Towards Instruction-Aligned Multimodal Generation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OmniGen2, a unified multimodal generator (Qwen2.5-VL-3B autoregressive backbone plus a diffusion decoder) with Omni-RoPE positional encoding, and aligns it with a three-stage multi-task Flow-GRPO curriculum (Edit, GenEval, IC), plus a new OmniContext in-context generation benchmark.

Problem

Open-source multimodal generators lack robust, generalized instruction alignment across text-to-image, editing, and in-context generation, and there is no standard benchmark for multi-subject in-context generation.

Approach

Condition a randomly initialized diffusion transformer on variable-length VLM text-token hidden states plus VAE features, use a 3D instance-identity positional encoding (Omni-RoPE), and align via a progressive sequence of single-task-reward RL stages (edit with EditScore, T2I with GenEval, IC with Qwen2.5-VL-72B reward) rather than joint training.

Key results

• OmniGen2 reaches 0.95 overall on GenEval, above BAGEL (0.88 with rewriter, 0.82 without) and Qwen-Image (0.91).
• On OmniContext, OmniGen2 scores 7.95 overall average, above Qwen-Image-Edit-2509 and other open models, though below GPT-4o (8.80).
• On GEdit-Bench-EN, OmniGen2 scores 7.21 overall, outperforming Gemini-2.5-Flash-Image (7.10) and trailing only Qwen-Image-Edit-2509 (7.54) among open models.

Surprising findings

• Human-preference reward (HPSv3) causes reward hacking: PQ rises to 8.22 while semantic consistency and in-context scores collapse.
• Training order matters: placing IC before GenEval lowers GEdit overall from 7.21 to 7.06.

Limitations

• Weaker performance on Chinese prompts and low-quality or downsampled input images.
• Struggles to modify human body shapes and with ambiguous multi-image instructions.
• Occasionally fails to faithfully reproduce objects from context images in in-context generation.

Experimental setup

Datasets: LLaVA-OneVision SEED-Data-Edit OmniEdit Echo-4o Recap-DataComp SAM-LLaVA ShareGPT4V LAION-Aesthetic JourneyDB BLIP3-o UltraEdit PromptFix

Benchmarks: MMBench MMMU MM-Vet GenEval OneIG-Bench ImgEdit-Bench GEdit-Bench-EN OmniContext Emu-Edit

Metrics: GenEval overall OneIG overall GEdit SC/PQ/Overall OmniContext PF/SC/Overall CLIP-I CLIP-Out DINO

Baselines: Qwen-Image-Edit-2509 BAGEL Gemini 2.5 Flash Image UniWorld-V1 OmniGen UNO UMO Step1X-Edit

Builds on: Flow-GRPO Lumina-Image 2.0 MetaQuery RoPE Rectified Flow

Closest papers by meaning

• DuoGen: Towards Autonomous Interleaved Multimodal Generation (CVPR 2026 · similarity 0.90)
• OmniGen: Unified Image Generation (CVPR 2025 · similarity 0.90)
• Instruct-Imagen: Image Generation with Multi-modal Instruction (CVPR 2024 · similarity 0.89)
• Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models (CVPR 2026 · similarity 0.88)
• Breaking Dual Bottlenecks: Evolving Unified Multimodal Models into Self-Adaptive Interleaved Visual Reasoners (ICML 2026 · similarity 0.88)
• Generative Multimodal Models are In-Context Learners (CVPR 2024 · similarity 0.88)