Research map / Vision-language and multimodal / Multimodal LLMs
VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VisionLLM v2, an end-to-end MLLM that connects a Vicuna-7B LLM to task-specific decoders (Grounding DINO, UniPose, Stable Diffusion, InstructPix2Pix) via "super link" routing tokens and learnable queries, trained jointly on hundreds of vision tasks in three stages.
Problem
Current MLLMs output only text, limiting structured and visual outputs such as detection, pose estimation, segmentation and image generation; existing tool-connecting methods are not end-to-end and naive embedding links do not scale to many tasks with conflicting training signals.
Approach
A "super link" couples routing tokens (e.g., [DET], [POSE], [SEG], [GEN]) added to the LLM vocabulary with task-bound learnable super-link queries appended after them; the queries' final hidden states condition task-specific decoders, so gradients flow back to the MLLM and tasks do not share queries.
Key results
- On COCO instance segmentation with Swin-T, the generalist model reaches 47.8 AP mask, close to specialist Grounding-DINO-T and above prior ResNet50-based generalists.
- VisionLLM v2-Chat reaches 82.9% Q->AR accuracy on VCR without task-specific fine-tuning, +3.5 over ASMv2.
- VisionLLM v2-Chat surpasses LLaVA-NeXT-7B by +9.7 on MMBench-EN and +7.0 on MMBench-CN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase the number of super-link queries from 1 to 8 (ablation) | VisionLLM v2 · visual perception tasks | performance across tasks | Helped |
| use separate rather than shared super-link queries for each decoder (ablation) | VisionLLM v2 · COCO box and keypoint prediction | AP | Helped |
| replace super-link queries with token embeddings (ablation) | VisionLLM v2 · COCO interactive segmentation | mIoU: 44.8 -> 43.7 | Hurt |
| replace three-stage training with one-stage training (ablation) | VisionLLM v2 · TextVQA | score: 66.2 -> 53.2 | Hurt |
| replace three-stage training with one-stage training (ablation) | VisionLLM v2 · COCO instance segmentation | APm: 47.6 -> 44.6 | Hurt |
| replace three-stage training with one-stage training (ablation) | VisionLLM v2 · COCO-Pose | AP: 74.2 -> 74.1 | No clear effect |
| train on image VQA alone (ablation) | VisionLLM v2 multi-task model · image VQA, instance segmentation, and image generation | task loss: -0.01; -0.11; -0.04 | Helped |
| train on instance segmentation alone (ablation) | VisionLLM v2 multi-task model · image VQA, instance segmentation, and image generation | task loss: +0.04; -0.12; + 0.19 | Mixed |
| train on image generation alone (ablation) | VisionLLM v2 multi-task model · image VQA, instance segmentation, and image generation | task loss: +0.03; +0.02; -0.04 | Mixed |
| unfreeze the region encoder during task fine-tuning (ablation) | VisionLLM v2 · COCO interactive segmentation | mIoU: 49.1 -> 65.4 | Helped |
Surprising findings
- Training on instance segmentation or image generation alone increases loss on other tasks, indicating multi-task conflict, while image VQA training helps all tasks.
- Shared super-link queries across decoders hurt keypoint AP and box AP compared with decoupled queries.
Limitations
- Three-stage training pipeline is relatively complex.
- Integration of downstream tools has only been preliminarily validated.
- Region encoder frozen in stage 3 limits interactive segmentation performance until fine-tuned.
Experimental setup
Datasets: ShareGPT4V All-Seeing VQAv2 COCO ADE20K RefCOCO/+/g LAION-Aesthetics MMIC
Benchmarks: VQAv2 GQA VizWiz ScienceQA TextVQA POPE MME MMBench SEED-Bench COCO LVIS PACO
Metrics: accuracy AP mAP CIDEr METEOR gIoU cIoU mIoU PCK@0.2 F-measure
Baselines: InstructBLIP LLaVA-NeXT Qwen-VL-Chat InternVL Shikra VisionLLM GLEE-Plus Hulk UniPose SEEM PSALM LISA
Builds on: LLaVA Grounding DINO UniPose Stable Diffusion InstructPix2Pix
Closest papers by meaning
- EfficientLLaVA: Generalizable Auto-Pruning for Large Vision-language Models (CVPR 2025 · similarity 0.90)
- CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding (ICLR 2024 · similarity 0.90)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- Visual Anchors Are Strong Information Aggregators For Multimodal Large Language Model (NeurIPS 2024 · similarity 0.90)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.