Research map / Vision-language and multimodal / Multimodal LLMs

Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents

NeurIPS 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Bifrost-1 bridges a frozen pretrained MLLM (Qwen2.5-VL) and FLUX.1-dev diffusion model by having the MLLM predict masked patch-level CLIP image embeddings, which condition the diffusion model through a lightweight latent ControlNet.

Problem

Integrating high-fidelity image generation into MLLMs without degrading their reasoning, at much lower training cost than single-architecture unified models or LLM-to-diffusion bridges that rely on 1D query tokens or text.

Approach

Use patch-level CLIP image embeddings, which are natively aligned with the MLLM's own visual encoder, as the latent bridge: a parallel visual generation branch (initialized from MLLM weights, with the MLLM frozen) predicts masked patch embeddings via MAR-style masked autoregression, and a latent ControlNet injects them into FLUX.1-dev.

Key results

• On ImageNet 256x256 (10K samples), Bifrost-1 reaches FID 25.77 vs 51.20 for the FLUX.1-dev backbone and 76.32 for a cross-attention variant.
• Bifrost-1 with Qwen2.5-VL 7B (62M images) scores GenEval 0.81 and DPG-Bench 77.67, and with 3B (9M images) GenEval 0.61 and DPG-Bench 76.41; MJHQ FID 16.21 for 7B.
• Understanding benchmarks match MetaQuery-XL exactly for Qwen2.5-VL 7B (MMMU 58.6, MM-Vet 66.6) since the MLLM stays frozen.

Measured effects reported in the paper

Surprising findings

• Bifrost-1 has worse COCO FID (34.35 for 7B) than several baselines despite competitive GenEval, which the authors attribute to FLUX.1-dev's aesthetic fine-tuning.
• Using 2D learnable query tokens or SigLIP latents performed far worse than CLIP latents (FID 118.69 and 274.16 vs 25.77).

Limitations

• Performance depends on the quality of the diffusion backbone (FLUX.1-dev), which inherits its limitations on rare or unseen scenes.
• Does not yet surpass SoTA unified models in generation performance given the much smaller compute and data budget.
• Trained on relatively small datasets (ImageNet, up to 62M pairs) compared with SoTA baselines.

Experimental setup

Datasets: ImageNet BLIP3-o-Pretrain-Long-Caption MSCOCO SA1B

Benchmarks: ImageNet 256x256 MJHQ-30K MSCOCO 30k GenEval DPG-Bench MME-P MMB SEED MMMU MM-Vet

Metrics: FID sFID IS rFID SSIM PSNR LPIPS GenEval score DPG-Bench score

Baselines: MetaQuery BLIP3-o JanusPro Janus JanusFlow TokenFlow-XL Transfusion LMFusion MetaMorph Show-o EMU3 Chameleon

Builds on: ControlNet MAR MetaQuery LMFusion unCLIP

Closest papers by meaning

• Generative Multimodal Pretraining with Discrete Diffusion Timestep Tokens (CVPR 2025 · similarity 0.91)
• I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models (ICML 2025 · similarity 0.90)
• Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.90)
• Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.90)
• LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.90)
• Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.89)