Research map / Vision-language and multimodal / Multimodal LLMs

DuoGen: Towards Autonomous Interleaved Multimodal Generation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds DuoGen, an interleaved text-image generator that couples a pretrained MLLM (Qwen2.5-VL-7B) with a video-pretrained DiT (Cosmos-Predict 2.5, 2B) via a two-stage decoupled training, using a 298k curated instruction dataset and a new interleaved benchmark.

Problem

General-purpose interleaved text-image generation is limited by scarce high-quality instruction-tuning data and base model capacity, and existing unified models need costly unimodal pretraining or cannot condition on multiple images.

Approach

Fuse a pretrained MLLM (text and <BOV> trigger) with a video-pretrained DiT that conditions on all prior images as frames, training first the MLLM on curated interleaved conversations and then freezing it to align the connector and DiT on interleaved context data, avoiding unimodal pretraining.

Key results

• On CoMM, DuoGen reaches IRS 7.76 versus 2.71 for the second-best method (MiniGPT-5), about 2.8x.
• On GenEval, DuoGen reaches 0.88 overall, the highest among the compared models in the table (Uni-CoT 0.83, Bagel 0.82).
• On Cooking-200 and How-to-500, DuoGen beats all open-source models and approaches Nano Banana on some metrics, e.g. Cooking-200 image-text coherence 4.75 vs 4.83 for Nano Banana.

Surprising findings

• Adding interleaved alignment data too early could harm the pretrained MLLM, so the MLLM is trained first and frozen during alignment.

Limitations

• Relies on GPT-4o/GPT-5 as judges, which can miss fine-grained artifacts; open-source judge results only in supplementary.
• Image resolution capped at 480 px for MLLM input to avoid OOM.
• Gap remains with Nano Banana on How-to-500 and on several image editing metrics.

Experimental setup

Datasets: CoMM ShareGPT-4o-Image NHR-Edit OmniGen1&2 UniWorld-V1 Echo-4o MM-Food-100k Instructables StoryBird eHow

Benchmarks: CoMM InterleavedBench Cooking-200 How-to-500 GenEval ImgEdit GEdit-Bench EN

Metrics: text completeness image completeness image coherence image quality image-text coherence IRS Elo GenEval overall ImgEdit overall GEdit SC/PQ/O

Baselines: Nano Banana SEED-LLaMA MiniGPT-5 Zebra-CoT Emu2 GILL Bagel OmniGen2 Uni-CoT Qwen-Image-Edit

Builds on: Qwen2.5-VL Cosmos-Predict 2.5 Bagel CoMM

Closest papers by meaning

• WeGen: A Unified Model for Interactive Multimodal Generation as We Chat (CVPR 2025 · similarity 0.92)
• Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization (NeurIPS 2025 · similarity 0.91)
• OmniGen2: Towards Instruction-Aligned Multimodal Generation (CVPR 2026 · similarity 0.90)
• Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Training (CVPR 2026 · similarity 0.90)
• OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.90)
• Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models (CVPR 2026 · similarity 0.90)