Research map / Vision-language and multimodal / Multimodal LLMs
NExT-GPT: Any-to-Any Multimodal LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents NExT-GPT, an end-to-end any-to-any multimodal LLM that connects Vicuna-7B to ImageBind encoders and diffusion decoders via lightweight projection layers, trained with a new 5K-dialogue modality-switching instruction dataset (MosIT).
Problem
Existing multimodal LLMs only understand multimodal inputs and generate text, while pipeline-style any-to-any systems pass discrete text between tools, causing error propagation and limiting end-to-end understanding and generation across text, image, video, and audio.
Approach
Connect a frozen-ish LLM to frozen multimodal encoders (ImageBind) and frozen diffusion decoders (SD, AudioLDM, Zeroscope) through trainable input/output projection layers, with LLM-emitted modality signal tokens conditioning the decoders, plus a multi-turn modality-switching instruction tuning dataset.
Key results
- NExT-GPT (Vicuna-7B) reports 123.7 NoCaps, 84.5 Flickr30K and 124.9 COCO CIDEr zero-shot, the best among compared models in that table.
- On text-to-image generation NExT-GPT reaches FID 10.07 on MS COCO versus 11.21 for SD-1.5 and 11.26 for CoDi.
- On MSR-VTT video captioning NExT-GPT reports 58.8 B@4 and 39.6 METEOR, with fine-tuning-based numbers marked separately.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NExT-GPT rather than InstructBLIP | image captioning · NoCaps | CIDEr: 123.1 -> 123.7 | Helped |
| Use NExT-GPT rather than Video-LLaVA | image question answering · VQA v2 | accuracy: 74.7 -> 66.7 | Hurt |
| Use NExT-GPT rather than Video-LLaVA | image question answering · VizWiz | accuracy: 48.1 -> 48.4 | Helped |
| Use NExT-GPT rather than Video-LLaVA | video question answering · MSVD-QA | accuracy: 70.7 -> 64.5 | Hurt |
| Use NExT-GPT rather than Video-LLaMA | video question answering · NExTQA | accuracy: 29.6 -> 50.7 | Helped |
| Use NExT-GPT rather than Codi | text-to-image generation · MS COCO | FID: 11.26 -> 10.07 | Helped |
| Use NExT-GPT rather than Codi | text-to-audio generation · AudioCaps | FAD: 1.80 -> 1.68 | Helped |
| Replace the grouping mechanism with a linear input layer (ablation) | NExT-GPT · VQA v2 | accuracy: 66.7 -> 63.8 | Hurt |
| Replace the grouping mechanism with a linear input layer (ablation) | NExT-GPT · VizWiz | accuracy: 48.4 -> 45.4 | Hurt |
| Replace the grouping mechanism with a linear input layer (ablation) | NExT-GPT · MSVD-QA | accuracy: 64.5 -> 60.8 | Hurt |
| Replace the grouping mechanism with a linear input layer (ablation) | NExT-GPT · MSRVTT-QA | accuracy: 61.4 -> 57.1 | Hurt |
| Replace the grouping mechanism with Q-former + linear layer (ablation) | NExT-GPT · VQA v2 | accuracy: 66.7 -> 65.1 | Hurt |
Surprising findings
- Simple linear input projection lowers perception scores substantially compared with the grouping mechanism (e.g., VQAv2 63.8 vs 66.7).
Limitations
- Supports only four modalities (text, image, video, audio) and relies on a single 7B Vicuna LLM.
- Generation quality is bounded by the frozen diffusion decoders, and the MosIT dataset is small (5K dialogues).
- Can produce hallucinated or low-quality content, as acknowledged in the impact statement.
Experimental setup
Datasets: CC3M WebVid-2M AudioCaps LLaVA-150K VideoChat Cleaned-Alpaca COCO-caption
Benchmarks: NoCaps Flickr30K COCO captioning VQAv2 VizWiz OKVQA MMBench SEED-Bench MSRVTT MSVD-QA MSRVTT-QA NExTQA
Metrics: CIDEr FID FAD CLIPSIM accuracy
Baselines: CoDi InstructBLIP LLaVA Video-LLaVA DreamLLM Emu UIO-2XXL GILL
Builds on: ImageBind Vicuna CoDi LoRA Stable Diffusion
Closest papers by meaning
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.91)
- Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception (CVPR 2024 · similarity 0.88)
- TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.88)
- MSP: Modality Self-Play for Training Multimodal LLMs Without Paired Cross-Modal Data (NeurIPS 2026 · similarity 0.88)
- L-Man: A Large Multi-modal Model Unifying Human-centric Tasks (AAAI 2025 · similarity 0.88)
- From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.