Research map / Vision-language and multimodal / Multimodal LLMs

LaTtE-Flow: Layerwise Timestep-Expert Flow-based Transformer

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LaTtE-Flow, a unified VLM-based image generation/understanding model whose transformer layers are split into timestep-specialized expert groups plus timestep-conditioned residual attention, achieving about 6x faster sampling than Janus Pro on ImageNet while keeping Qwen2-VL-2B understanding performance.

Problem

Unified multimodal models that generate images with diffusion/flow matching need many full-backbone forward passes per image, and they require extensive pretraining, making generation slow for real-time or resource-constrained use.

Approach

Partition the transformer layers into K disjoint groups, each responsible for a timestep interval of the flow-matching trajectory, so only M=L/K layers run per step; add a per-head, timestep-gated residual connection of image self-attention maps across adjacent layers.

Key results

• LaTtE-Flow reaches FID 5.79 and IS 213.1 on ImageNet with 40 steps, 500M activated params per step, and 0.052 s/img versus 0.311 s/img for Janus Pro (about 6x faster).
• Removing timestep-conditioned residual attention raises FID from 5.79 to 8.26 and lowers IS from 213.1 to 157.0.
• Understanding results match the frozen Qwen2-VL-2B backbone exactly (e.g., MMBench 74.9, POPE 87.3), since the understanding path is unchanged.

Surprising findings

• The understanding scores are identical to the frozen Qwen2-VL-2B backbone, so the method adds no understanding gains and inherits the backbone's capabilities.

Limitations

• Trained for only 240K steps, much less than existing unified models, and evaluated only on ImageNet class-conditioned captions.
• Uniform timestep partitioning and layer grouping are heuristic; optimal partitioning is left open.
• Understanding performance is simply inherited from the frozen backbone, so no understanding improvement is shown.

Experimental setup

Datasets: ImageNet

Benchmarks: ImageNet 256x256 generation (FID, IS, Precision, Recall) MMBench SEED POPE MM-Vet MME-P MMMU RealWorldQA TextVQA

Metrics: FID Inception Score Precision Recall inference time per image accuracy on understanding benchmarks

Baselines: Show-o Janus Pro LMFusion-style baseline (LaTtE-Flow-Couple) ADM DiT-XL/2 MaskGIT VQGAN EMU2 Chat VILA-U EMU3 MetaMorph Seed-X

Builds on: Qwen2-VL LMFusion Transfusion Conditional Flow Matching DiT DC-AE

Closest papers by meaning

• LaVin-DiT: Large Vision Diffusion Transformer (CVPR 2025 · similarity 0.90)
• JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation (CVPR 2025 · similarity 0.87)
• Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.87)
• LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.87)
• Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens (CVPR 2025 · similarity 0.87)
• To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Models (ICLR 2026 · similarity 0.87)