Research map / Vision-language and multimodal / Multimodal LLMs

Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UniMRG, a post-training method where unified multimodal models learn to generate pixel reconstructions, depth maps, and segmentation maps alongside understanding SFT, improving understanding (MMVP, HallusionBench, VSR) and generation across Harmon, OpenUni, and Show-o.

Problem

Existing UMM post-training mainly uses understanding to improve generation; whether generation can improve visual understanding (fine-grained perception, hallucination, spatial reasoning) is largely unexplored.

Approach

Train UMMs to generate intrinsic image representations (pixel reconstruction, depth from Depth Anything V2, segmentation from SAM masks) jointly with standard VQA loss, so generation of dense geometric and structural cues transfers to understanding; architecture-agnostic with no inference changes.

Key results

• On OpenUni-3.6B, UniMRG raises VSR from 66.69 to 73.90 (+7.21), HallusionBench from 60.88 to 64.56, and MMVP from 71.67 to 74.67.
• On Harmon-1.5B, UniMRG improves GenEval from 71.37 to 85.26 and understanding on RWQA from 46.67 to 51.90, while SFT alone collapses GenEval to 0.30.
• Depth generation similarity on MidjourneyV6 rises from 0.623 to 0.822 for Harmon and 0.617 to 0.834 for OpenUni; Show-o gains only marginally (0.637 to 0.664).

Surprising findings

• Understanding-only SFT causes generation to collapse (Harmon GenEval 71.37 to 0.30), showing generation capability is fragile under understanding-only post-training.
• Show-o shows only marginal understanding gains, which the authors attribute to its 4,096-token VQ codebook limiting joint multi-representation generation.

Limitations

• Gains on Show-o are small, attributed to a limited VQ codebook capacity.
• Only depth and segmentation are studied; other intrinsic representations are left for future work.
• Depth and segmentation targets come from external teachers (Depth Anything V2, SAM), so quality is bounded by them.

Experimental setup

Datasets: LLaVA Mix-665K LLaVA-Next-Data MidjourneyV6

Benchmarks: MMBench MMVP HallusionBench RealWorldQA VSR GenEval DPGBench

Metrics: accuracy overall score 1-MAE

Baselines: SFT RecA Janus-Pro-7B Harmon-1.5B Show-o-1.3B BAGEL UniLIP InternVL3-2B Qwen2.5-VL-3B Qwen3-VL-2B

Builds on: RecA MetaQueries Depth Anything V2 Segment Anything

Closest papers by meaning

• Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (CVPR 2025 · similarity 0.91)
• Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models (CVPR 2026 · similarity 0.90)
• RealUnify: Do Unified Models Truly Benefit from Unification? A Comprehensive Benchmark (CVPR 2026 · similarity 0.90)
• Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (ICLR 2025 · similarity 0.89)
• MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.89)
• Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.89)