Research map / Vision-language and multimodal / Multimodal LLMs

LatentUM: Unleashing the Potential of Interleaved Cross-Modal Reasoning via a Latent-Space Unified Model

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LatentUM, a unified autoregressive model that discretizes CLIP features via model-behavior-aligned quantization so that interleaved cross-modal reasoning happens in a shared semantic latent space without pixel-space mediation.

Problem

Existing unified models use disjoint or pixel-mediated visual representations for understanding and generation, which hampers interleaved cross-modal reasoning (visual planning, self-reflection, world modeling) and causes codec bias and modality misalignment.

Approach

Quantize CLIP features into discrete semantic tokens by training the quantizer to preserve a VLM's output distribution (KL loss) rather than reconstruct pixels; pair this with a Mixture-of-Modal Experts transformer (separate understanding/generation branches sharing self-attention) and a separately trained diffusion pixel decoder used only for visualization.

Key results

• LatentUM-Base reaches GenEval overall 0.85, and LatentUM-Vis-Gen with self-reward reaches 0.87 (0.92 with pixel reward), best among unified models reported.
• LatentUM-Vis-Plan (fine-grained) reaches 0.99 average accuracy on VSP versus 0.76 for ThinkMorph.
• LatentUM-Base scores MMMU 54.6 and MMBench 80.3 with continuous features, staying competitive with unified baselines.

Surprising findings

• Self-reward from the model's own understanding branch improves GenEval from 0.85 to 0.87 without any external reward model.
• Pixel decoder works with CFG scale 1.0 (no unconditional guidance), and the model produces coherent images without CFG.

Limitations

• Fixed-resolution generation (448x448) and modest pretraining scale.
• World-model rollout still re-renders pixels between steps, so it is not fully latent.
• MBAQ is aligned to a single VLM's behavior; generality of the semantic representation is underexplored.

Experimental setup

Datasets: LLaVA-v1.5-665K BLIP3o 32M text-image pairs ThinkMorph VSP data RECON GenEval prompts GenEval2 prompts

Benchmarks: MME-P POPE SEED MMBench MMMU GenEval GenEval2 VSP NWM

Metrics: accuracy overall GenEval score TIFA GM/AM ATE RPE

Baselines: Bagel Janus Pro Show-o2 BLIP3o ThinkMorph Transfusion-RAE Mirage MVoT NWM VQVAE (LlamaGen)

Builds on: Mixture-of-Transformers InternVL3.5 GRPO multi-codebook quantization BLIP3o

Closest papers by meaning

• ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation (ICLR 2026 · similarity 0.91)
• Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Training (CVPR 2026 · similarity 0.89)
• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.88)
• UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMs (NeurIPS 2026 · similarity 0.88)
• LatentOmni: Rethinking Omni-Modal Understanding via Unified Audio-Visual Latent Reasoning (NeurIPS 2026 · similarity 0.88)
• Monet: Reasoning in Latent Visual Space Beyond Image and Language (CVPR 2026 · similarity 0.88)