Research map / Vision-language and multimodal / Multimodal LLMs
LMFusion: Adapting Pretrained Language Models for Multimodal Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts a frozen text-only Llama-3 8B into a multimodal generator by adding parallel image-specific transformer modules trained with the Transfusion diffusion objective, and extends the recipe to LLaVA-NeXT (LLaVAFusion).
Problem
Training unified text-image generative models from scratch is computationally expensive and degrades text-only performance, so the paper asks how to add image understanding and generation to pretrained text-only LLMs while preserving their language abilities.
Approach
Modality-specific QKV projections, output projections, and FFNs (deep separation) let the frozen Llama-3 text pathway remain intact while a parallel image pathway, initialized from Llama-3 weights and trained with diffusion loss, shares only the self-attention computation across modalities.
Key results
- LMFusion at 0.5x FLOPs achieves CIDEr 38.3 vs Transfusion 32.0 on MS-COCO image captioning while matching Llama-3 on HellaSwag (60.0), SIQA (48.1), and WinoGrande (72.8).
- Transfusion scores 51.0 / 42.3 / 64.3 on HellaSwag / SIQA / WinoGrande, versus Llama-3's 60.0 / 48.1 / 72.8.
- LLaVAFusion reports MMMU 41.7, ChartQA 69.5, RealWorldQA 60.0, MME-P 1603.7, and FID 8.2 among compared unified models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use frozen, modality-separated image modules initialized from Llama-3 instead of training Transfusion from scratch | LMFusion · MS-COCO test | CIDEr: 32.0 -> 38.3 | Helped |
| Use LMFusion with 0.5× FLOPs instead of Transfusion | LMFusion · HellaSwag, SIQA, and WinoGrande | accuracy: 51.0 -> 60.0; 42.3 -> 48.1; 64.3 -> 72.8 | Helped |
| Use LMFusion with 0.5× FLOPs instead of Transfusion | LMFusion · MS-COCO generation | FID without CFG: 14.4 -> 13.9 | Helped |
| Use LMFusion with 0.5× FLOPs instead of Transfusion | LMFusion · MS-COCO generation | FID with CFG: 8.70 -> 8.75; 8.70 -> 8.61 | Mixed |
| Use LMFusion instead of Transfusion | LMFusion · MS-COCO generation | CLIP score: 22.1 -> 22.0; 24.4 -> 24.4 | No clear effect |
| Use modality-specific FFNs and attention instead of a shared dense transformer (ablation) | LMFusion · Image understanding and generation | performance | Helped |
| Add modality-specific attention to modality-specific FFNs (ablation) | LMFusion · Image understanding and generation | performance | Helped |
| Use a text-to-image learning-rate ratio of 0.1 instead of 1 in a dense model (ablation) | Dense Llama-3 8B · Language-only benchmarks | performance gap from original Llama-3: 7% to 2% | Helped |
| Use a text-to-image learning-rate ratio of 0.1 instead of 1 in a dense model (ablation) | Dense Llama-3 8B · Image understanding and generation | performance | Hurt |
| Freeze text components instead of fully fine-tuning them in a dense model (ablation) | Dense Llama-3 8B · Image understanding and generation | performance | Hurt |
| Freeze text modules in the dense model during multimodal training (ablation) | Dense Llama-3 8B · HellaSwag | accuracy: 15%; 7% | Hurt |
| Separate modality-specific FFNs and attention instead of sharing dense transformer components (ablation) | LMFusion · Image understanding and generation | performance | Helped |
Surprising findings
- Naive finetuning of dense Llama-3 on multimodal data causes about a 15% initial HellaSwag drop and a persistent gap of roughly 7% that never recovers.
- Freezing text modules in a dense model hurts image understanding and generation, whereas in the deep-separation design freezing text modules improves both.
Limitations
- Built on text-only LLMs; the image pathway has no text-only data in training and depends on Transfusion-style data and recipe.
- Image generation at 256x256 resolution with a 380M proprietary Shutterstock dataset; the data is not released.
- Image understanding and generation gains are reported against a Transfusion model trained from scratch in a controlled setting, with limited comparison to other frontier systems.
Experimental setup
Datasets: Shutterstock image-caption data (380M) MS-COCO MagicBrush HellaSwag PIQA SIQA WinoGrande
Benchmarks: HellaSwag SIQA WinoGrande MS-COCO CIDEr MS-COCO FID MS-COCO CLIP MMMU ChartQA RealWorldQA MME-P
Metrics: accuracy CIDEr FID CLIP score relative FLOPs
Baselines: Transfusion Llama 3 EMU-3 Show-O Janus Chameleon MetaMorph LLaVA-NeXT
Builds on: Transfusion Llama-3 LLaVA-NeXT Mixture of Experts
Closest papers by meaning
- MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer (ICLR 2026 · similarity 0.92)
- Elucidating the design space of language models for image generation (ICML 2025 · similarity 0.92)
- The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.91)
- Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representations (NeurIPS 2025 · similarity 0.91)
- A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation (CVPR 2025 · similarity 0.91)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.