Research map / Vision-language and multimodal / Human and embodied modeling

JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation

CVPR 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Vision-language pretraining · 20 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes JanusFlow, a unified 1.3B LLM that performs image understanding via next-token prediction and image generation via rectified flow in SDXL-VAE latent space, with decoupled understanding/generation encoders and representation alignment.

Problem

Unifying multimodal understanding and text-to-image generation in a single LLM without the architectural complexity of external diffusion tools or the limits of VQ autoregressive tokenization.

Approach

Train rectified flow directly inside an LLM with only lightweight encoder/decoder layers, decouple SigLIP understanding encoder from ConvNeXt generation encoder/decoder, and align intermediate LLM features with the understanding encoder features during generation training.

Key results

• Achieves GenEval 0.63 and DPG-Bench 80.09% and MJHQ FID-30k 9.51 for image generation, best FID among 1.3B models.
• Scores 74.9 on MMBench, 70.5 on SEED-Bench and 60.3 on GQA, exceeding LLaVA-v1.5 and Qwen-VL-Chat on these benchmarks.
• Ablation shows representation alignment lowers MJHQ FID-10k from 19.84 to 17.61 and raises CLIP score from 24.94 to 26.40.

Measured effects reported in the paper

Surprising findings

• Causal attention suffices for the rectified flow generation pass; alternative attention masks gave no benefit in preliminary experiments.
• Shared encoders for understanding and generation (Exp. B) substantially hurt understanding performance relative to decoupled encoders.

Limitations

• Understanding performance on some benchmarks (e.g., MMMU 29.3, MM-Vet 30.9) trails some baselines.
• Generation relies on a fixed set of training stages and large in-house data (2 million images) that is not released.

Experimental setup

Datasets: LAION-Aesthetics ShareGPT4V DeepSeek-VL chart and table data DenseFusion-1M Megalith-10M DeepSeek-LLM text corpus Open Images SA-1B JourneyDB in-house data

Benchmarks: GenEval DPG-Bench MJHQ FID-30k POPE MME MMBench SEED-Bench VQAv2 GQA MM-Vet MMMU ChartQA

Metrics: FID GenEval overall score DPG-Bench overall score accuracy

Baselines: Janus Show-o Transfusion LLaVA-v1.5 Qwen-VL-Chat SDXL VILA-U DALL-E 2

Builds on: rectified flow Janus Transfusion SDXL-VAE REPA

Closest papers by meaning

• Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation (CVPR 2025 · similarity 0.91)
• STARFlow2: Bridging Language Models and Normalizing Flows for Unified Multimodal Generation (NeurIPS 2026 · similarity 0.90)
• JetFormer: An autoregressive generative model of raw images and text (ICLR 2025 · similarity 0.89)
• HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generation (NeurIPS 2025 · similarity 0.89)
• LMFusion: Adapting Pretrained Language Models for Multimodal Generation (NeurIPS 2025 · similarity 0.88)
• Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency (ICLR 2024 · similarity 0.88)