Research map / Image generation / Text-to-image diffusion
OmniGen-AR: AutoRegressive Any-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniGen-AR, a unified autoregressive any-to-image generator that handles text, spatial, and visual-context conditions via a shared visual tokenizer, with Disentangled Causal Attention as a training-time regularizer.
Problem
Autoregressive visual generators are limited to single-modality conditions (text or labels), and causal attention over condition tokens can leak shortcut information into content tokens in tasks like editing and frame prediction.
Approach
Disentangled Causal Attention (DCA) splits the causal mask so content tokens attend to text and preceding content but not condition tokens, applied randomly during training only; a shared visual tokenizer (Cosmos-DV8x16x16) discretizes all conditions, images and videos into one sequence model.
Key results
- Achieves 0.63 overall on GenEval at 1.5B parameters, outperforming other models under 1B, and competitive with larger models.
- Reaches 80.02 VBench total at 1.5B parameters, the first vanilla discrete-token AR model to exceed 80 on VBench per the authors.
- DCA with 10% replacement raises Emu-Edit CLIP text similarity from 0.15 to 0.20 on the 0.5B model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model size from 0.5B to 1.5B parameters | OmniGen-AR · GenEval | Overall: 0.57 to 0.63 | Helped |
| Increase model size from 0.5B to 1.5B parameters | OmniGen-AR · VBench | Total: 74.72 to 80.02 | Helped |
| Add DCA with 10% replacement probability (ablation) | OmniGen-AR · Emu-Edit test set | Emu-CT: 0.15 to 0.20 | Helped |
| Add DCA with 10% replacement probability (ablation) | OmniGen-AR · VBench | Total: 70.33 to 74.72 | Helped |
| Add DCA with 10% replacement probability (ablation) | OmniGen-AR · ADE20K segmentation-to-image | Mask: 24.76 to 25.33 | Helped |
| Train tasks jointly instead of separately (ablation) | OmniGen-AR · GenEval | GEval: 0.57 to 0.55 | Hurt |
| Train tasks jointly instead of separately (ablation) | OmniGen-AR · VBench | VBen: 77.18 to 74.72 | Hurt |
| Train tasks jointly instead of separately (ablation) | OmniGen-AR · Emu-Edit test set | Emu: 0.18 to 0.20 | Helped |
| Train tasks jointly instead of separately (ablation) | OmniGen-AR · ADE20K segmentation-to-image | Mask: 22.59 to 25.33 | Helped |
| Use DCA replacement probabilities ranging from 5% to 30% rather than 0% (ablation) | OmniGen-AR · ADE20K segmentation-to-image | Mask: 24.76; 25.78; 25.33; 25.16; 21.49 | Mixed |
Surprising findings
- Joint multi-task training degrades text-to-image and text-to-video benchmarks, though it improves editing and segmentation-to-image results.
Limitations
- Instruction-following failures, e.g., removing the wrong object in editing.
- Blurry or structurally inconsistent outputs for depth and segmentation conditions under sparse control signals.
- Depth-to-image and segmentation results lag some baselines (e.g., ControlAR, OmniGen on mIoU/RMSE).
Experimental setup
Datasets: CC3M CC12M OpenImages SAM1B Megalith-10M Panda70M HD-VILA-100M JourneyDB Synthetic-dataset-1M MagicBrush InstructPix2Pix SEED-Edit
Benchmarks: GenEval VBench Kinetics-600 Emu-Edit ADE20K MultiGen-Depth-Eval MagicBrush
Metrics: GenEval overall VBench total FVD CLIP text similarity CLIP image similarity mIoU RMSE token match ratio
Baselines: SDv1.5 PixArt-alpha SDv2.1 LlamaGen SimAR-SFT LDM DALL-E 2 Show-o Infinity Janus Emu3 CogVideo
Builds on: Qwen2.5 Cosmos tokenizer classifier-free guidance EditAR OmniGen
Closest papers by meaning
- Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation (NeurIPS 2025 · similarity 0.89)
- Parallelized Autoregressive Visual Generation (CVPR 2025 · similarity 0.89)
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.89)
- Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots (ICML 2025 · similarity 0.89)
- EditAR: Unified Conditional Generation with Autoregressive Models (CVPR 2025 · similarity 0.89)
- Autoregressive Video Generation without Vector Quantization (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.