Research map / Video / Video generation
UniReal: Universal Image Generation and Editing via Learning Real-world Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
UniReal unifies image generation and editing tasks by treating input and output images as frames of a discontinuous video, training a 5B diffusion transformer with full attention on video-derived and task-specific data.
Problem
Image generation and editing tasks (editing, customization, insertion, composition, perception) are typically handled by task-specific methods and data, limiting shared learning and generalization.
Approach
Treat varying numbers of input and output images as pseudo-video frames within one diffusion transformer using full attention, add index embeddings, image-role prompts and hierarchical context prompts, and use large-scale video frame pairs and object masks as universal supervision.
Key results
- On MagicBrush, UniReal achieves CLIPdir 0.151 and CLIPout 0.308, the best among compared methods in the table.
- On DreamBench customization, UniReal reaches CLIP-T 0.326 (best) with CLIP-I 0.806 and DINO 0.702, competitive with SuTI and BootPIG.
- User studies with 10 annotators show UniReal preferred over SuTI, OmniGen, UltraEdit and AnyDoor across detail preservation, alignment and quality (preference rates mostly 53-77%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove context prompt (ablation) | UniReal · MagicBrush test | CLIPdir: 0.144 -> 0.151 | Hurt |
| remove context prompt (ablation) | UniReal · MagicBrush test | CLIPout: 0.294 -> 0.308 | Hurt |
| remove context prompt (ablation) | UniReal · DreamBench | CLIP-T: 0.315 -> 0.326 | Hurt |
| remove image prompt (ablation) | UniReal · MagicBrush test | CLIPdir: 0.136 -> 0.151 | Hurt |
| remove image prompt (ablation) | UniReal · DreamBench | CLIP-T: 0.295 -> 0.326 | Hurt |
| train on expert data only instead of multi-task full data (ablation) | UniReal · MagicBrush test | CLIPdir: 0.139 -> 0.151 | Hurt |
| train on expert data only instead of multi-task full data (ablation) | UniReal · DreamBench | CLIP-T: 0.309 -> 0.326 | Hurt |
| use multi-task full training instead of expert data only (ablation) | UniReal · MagicBrush test | DINO: 0.788 -> 0.837 | Mixed |
| use multi-task full training instead of expert data only (ablation) | UniReal · DreamBench | DINO: 0.708 -> 0.702 | No clear effect |
Surprising findings
- A model trained only on single-input video frame pairs can handle some multi-input tasks such as reference-based object insertion, though unstable.
- Multi-task full training outperforms task-specific expert-data-only training on the ablated metrics.
Limitations
- Stability decreases and computation becomes intensive when the number of images exceeds five.
- Video-only training sometimes fails to precisely follow instructions and cannot cover subtasks such as stylization.
- Relies on in-house datasets (300M T2I, 2M instruct editing, 100K object insertion) that are not public.
Experimental setup
Datasets: InstructPix2Pix UltraEdit VTON-HD RefCOCO DreamBench MagicBrush EMU Edit Video Frame2Frame (constructed) Video Multi-object (constructed) Video Object Insertion (constructed) Video ObjectAdd (constructed) Video SEG (constructed)
Benchmarks: EMU Edit test set MagicBrush test set DreamBench
Metrics: CLIPdir CLIPim CLIPout L1 DINO CLIP-T CLIP-I preference rate
Baselines: InstructPix2Pix MagicBrush UltraEdit OmniGen EMU Edit ACE PixWizard AnyDoor BLIP-Diffusion ELITE Emu2 SuTI
Builds on: Sora-style video diffusion transformers flow matching SAM2 Kosmos-2 T5
Closest papers by meaning
- WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perception (NeurIPS 2025 · similarity 0.87)
- CAGE: Unsupervised Visual Composition and Animation for Controllable Video Generation (AAAI 2025 · similarity 0.87)
- Frame In-N-Out: Unbounded Controllable Image-to-Video Generation (NeurIPS 2025 · similarity 0.86)
- EditAR: Unified Conditional Generation with Autoregressive Models (CVPR 2025 · similarity 0.86)
- Generative Omnimatte: Learning to Decompose Video into Layers (CVPR 2025 · similarity 0.86)
- DynaTokens: Teaching Dynamics to Camera-Controlled Video Models at Test Time (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.