Research map / Vision-language and multimodal / Multimodal LLMs
Jointly Training Large Autoregressive Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fuses a 7B CM3leon text-to-image autoregressive model with a 7B text LLM via weight averaging, width concatenation, or bidirectional cross-attention, then continued pretraining and small-data instruction tuning for interleaved image-text generation.
Problem
Integrating a text-only LLM and an autoregressive text-to-image model into one robust model that can generate seamless interleaved text and image outputs.
Approach
Treats merged/fused parent models as an initialization for continued mixed-modal training rather than merging without training; proposes JAM-Cross, which inserts bidirectional cross-attention between the two parent transformers, and instruction-tunes on a small curated interleaved dataset (LIMA, wikiHow, Shutterstock) to learn interleaved generation.
Key results
- JAM-Cross (19B) reaches MS-COCO PPL 147.6, better than CM3leon 7B (149.0) and the other JAM variants (JAM-Uniform 177.5, JAM-Width 159.5).
- JAM-Cross achieves text commonsense results close to the parent LLM-4k (e.g., PIQA 75.4 vs 76.7; Winograd 81.0 vs 83.9), while JAM-Uniform and JAM-Width degrade sharply on text tasks (e.g., PIQA 62.4 and 57.8).
- Instruction tuning with Shutterstock pretraining data mixed in lowers MS-COCO PPL from 190.2 to 164.5 relative to instruction tuning without it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use bidirectional cross-attention fusion | JAM-Cross · MS-COCO validation | PPL: 149.0 -> 147.6 | Helped |
| use bidirectional cross-attention fusion | JAM-Cross · StoryCloze zero-shot | score: 79.3 -> 79.8 | Helped |
| use bidirectional cross-attention fusion | JAM-Cross · PIQA zero-shot | score: 76.7 -> 75.4 | Hurt |
| use bidirectional cross-attention fusion | JAM-Cross · ARC-C zero-shot | score: 45.9 -> 41.6 | Hurt |
| use bidirectional cross-attention fusion | JAM-Cross · ARC-E zero-shot | score: 67.7 -> 67.2 | Hurt |
| use bidirectional cross-attention fusion | JAM-Cross · Winograd zero-shot | score: 83.9 -> 81.0 | Hurt |
| use bidirectional cross-attention fusion | JAM-Cross · Winogrande zero-shot | score: 66.2 -> 66.0 | Hurt |
| copy backbone weights rather than average them when initializing new parameters (ablation) | JAM-Width · MS-COCO validation | PPL: 175.4 -> 159.5 | Helped |
| copy backbone weights rather than average them when initializing new parameters (ablation) | JAM-Width · Wikipedia | PPL: 9.0 -> 7.34 | Helped |
| insert cross-attention layers at frequency 2 instead of none (ablation) | JAM-Cross · Wikipedia | PPL: 7.86 -> 7.18 | Helped |
| insert cross-attention layers at frequency 2 instead of none (ablation) | JAM-Cross · MS-COCO validation | PPL: 153.2 -> 149.0 | Helped |
| use frequency 2 rather than frequency 1 for cross-attention layers (ablation) | JAM-Cross · Wikipedia | PPL: 7.53 -> 7.18 | Helped |
Surprising findings
- Weight averaging (JAM-Uniform) and width concatenation (JAM-Width) lose much text-only capability, and JAM-Width with copied weights outperforms averaged initialization on image-text PPL.
Limitations
- Limited to 7B parent models with identical architecture; scaling not studied.
- Context length of 4k tokens restricts outputs to roughly 2 to 3 images; typically one or two images generated.
- Struggles to generate images faithful to abstract concepts; relies on wiki-style single-turn QA.
Experimental setup
Datasets: Shutterstock image-text subset (30B tokens from CM3leon data) text corpora (CommonCrawl, C4, Wikipedia, Books3, arXiv; 30B tokens) LIMA 1k wikiHow (3000 articles) MS-COCO
Benchmarks: PIQA ARC-Challenge ARC-Easy StoryCloze Winograd Winogrande MS-COCO validation PPL
Metrics: zero-shot accuracy perplexity
Baselines: CM3 RA-CM3 CM3leon GILL LLaMA GPT-3 JAM-Uniform JAM-Width
Builds on: CM3leon CM3 LIMA Flamingo GILL
Closest papers by meaning
- Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities (CVPR 2024 · similarity 0.90)
- SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generation (CVPR 2025 · similarity 0.87)
- Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Heads (ICML 2025 · similarity 0.87)
- The Design Space of Tri-Modal Masked Diffusion Models (NeurIPS 2026 · similarity 0.87)
- JetFormer: An autoregressive generative model of raw images and text (ICLR 2025 · similarity 0.87)
- Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.