Research map / Vision-language and multimodal / Multimodal LLMs
CoDi-2: In-Context Interleaved and Interactive Any-to-Any Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
CoDi-2 is an MLLM that aligns text, image and audio via ImageBind features and autoregressively predicts output features decoded by diffusion models, enabling in-context interleaved any-to-any generation.
Problem
Existing multimodal generative models lack zero-shot in-context learning, multi-round interactive consistency, and the ability to follow modality-interleaved instructions for any-to-any generation.
Approach
Empower a Llama-2 LLM with ImageBind-aligned multimodal encoders and train it to predict the conditional features of the output modality, which are fed into synchronized diffusion decoders (Stable Diffusion, AudioLDM2, zeroscope), with an MSE feature loss plus diffusion and token losses, and build in-context interleaved training data including text-only repurposing.
Key results
- On DreamBench, the model reports DINO 0.703, CLIP-I 0.852, CLIP-T 0.311, above KOSMOS-G (0.694/0.847/0.287) and DreamBooth (0.668/0.803/0.305) in the table.
- On MS-COCO image editing, CLIPSIM is 0.147, below InstructPix2Pix (0.151) and above SDEdit-1/2T (0.134).
- On audio adding, dropping, and replacement tasks, the model achieves the lowest LSD, KL, and FD in all three settings (e.g., adding LSD 1.21, FD 19.72 vs AUDIT 1.35 and 21.80).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add MLLM-to-diffusion pipeline with original input features (MSE alignment on C_x(x)) and concatenated noise conditioning for fidelity | CoDi-2 image/audio editing · MS-COCO image editing | CLIPSIM | Mixed |
| Use MLLM (Llama-2 + ImageBind + diffusion decoders) with subject-driven zero-shot generation instead of per-subject fine-tuning | CoDi-2 subject-driven image generation · DreamBench | DINO / CLIP-I / CLIP-T: DINO 0.703 vs DreamBooth 0.668 (+0.035) | Helped |
| Use instruction-conditioned audio editing trained on AudioSet-derived add/drop/replace data instead of caption-based SDEdit | CoDi-2 audio editing · AudioSet editing (adding/dropping/replacement) | LSD, KL, FD: adding LSD 1.21 vs AUDIT 1.35; replacement FD 17.32 vs AUDIT 21.65 | Helped |
| Zero-shot in-context generation with interleaved instructions (no task finetuning) | CoDi-2 · qualitative in-context tasks (Tables 1-3 examples) | qualitative demonstrations | No clear effect |
Surprising findings
- On MS-COCO image editing, the model's CLIPSIM (0.147) is slightly below InstructPix2Pix (0.151), despite its gains on DreamBench and audio editing.
Limitations
- Visual concept learning is underrepresented in training data, per the authors' appendix.
- Image editing on MS-COCO CLIPSIM does not beat InstructPix2Pix.
- Many claimed zero-shot capabilities (e.g., reasoning, multi-round chat) are shown qualitatively rather than quantitatively evaluated.
Experimental setup
Datasets: MIMIC-IT LAION-400M AudioSet WebVid InstructPix2Pix Kosmos-G dataset Open Images V7 Alpaca DreamBench MS-COCO
Benchmarks: DreamBench MS-COCO AUDIT audio editing evaluation
Metrics: DINO CLIP-I CLIP-T CLIPSIM FID LSD KL FD
Baselines: DreamBooth Re-Imagen KOSMOS-G SDEdit InstructPix2Pix AUDIT
Builds on: CoDi ImageBind Llama 2 InstructPix2Pix Kosmos-G
Closest papers by meaning
- MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Models (ICLR 2026 · similarity 0.89)
- WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation (CVPR 2026 · similarity 0.89)
- Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization (NeurIPS 2025 · similarity 0.88)
- Generative Multimodal Models are In-Context Learners (CVPR 2024 · similarity 0.88)
- OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation (CVPR 2025 · similarity 0.88)
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.