Research map / Vision-language and multimodal / Human and embodied modeling

Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation

CVPR 2025 · Vision-language and multimodal › Human and embodied modeling · approach: Vision-language pretraining · 44 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Janus decouples visual encoding (SigLIP for understanding, VQ tokenizer for generation) feeding a single autoregressive DeepSeek-LLM 1.3B transformer, trained in three stages, for unified multimodal understanding and generation.

Problem

Unified multimodal models that use a single visual encoder for both understanding and generation suffer trade-offs, especially degraded multimodal understanding performance.

Approach

Decouple visual encoding into two independent pathways (a SigLIP semantic encoder for understanding and a VQ tokenizer for generation) that are both fed into one shared autoregressive transformer, letting each task use its own suitable encoder.

Key results

• Janus (1.3B) scores 87.0 POPE, 69.4 MMBench, 63.7 SEED, outperforming LLaVA-v1.5 (7B) on these benchmarks.
• Janus achieves 61% overall accuracy on GenEval, above Show-o (53%) and SDXL (55%).
• Janus reaches FID 8.53 on MSCOCO-30K, and in ablation the shared-encoder baseline (Exp-B) scores 52.7 MMB vs 69.4 for decoupled Janus.

Measured effects reported in the paper

Surprising findings

• Decoupled Janus is worse than a semantic-tokenizer-only shared baseline (Exp-B) on COCO-FID (8.53 vs 7.11), which the authors attribute partly to the larger parameter count of the shared encoder.

Limitations

• Visual generation FID on MSCOCO is weaker than some specialized generation-only models, and the decoupled model underperforms a shared semantic tokenizer on COCO-FID in ablation.
• Extensions to stronger encoders, high resolution, and new modalities are proposed but not evaluated.

Experimental setup

Datasets: ShareGPT4V ImageNet-1k DeepSeek-LLM pretraining corpus WikiHow WIT DeepSeek-VL table and chart data MJHQ-30K in-house image-text data

Benchmarks: VQAv2 GQA POPE MME SEED MMBench MM-Vet MMMU MSCOCO-30K MJHQ-30K GenEval

Metrics: accuracy FID GenEval overall accuracy

Baselines: Chameleon Show-o LWM VILA-U LLaVA-v1.5 Qwen-VL-Chat SDXL DALL-E 2 LlamaGen Emu SEED-X

Builds on: Chameleon LlamaGen LLaVA DeepSeek-LLM SigLIP

Closest papers by meaning

• Generation Enhances Understanding in Unified Multimodal Models via Multi-Representation Generation (ICML 2026 · similarity 0.91)
• JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation (CVPR 2025 · similarity 0.91)
• UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.91)
• Show-o: One Single Transformer to Unify Multimodal Understanding and Generation (ICLR 2025 · similarity 0.90)
• Learning to Generate via Understanding: Understanding-Driven Intrinsic Rewarding for Unified Multimodal Models (CVPR 2026 · similarity 0.89)
• Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.89)