Research map / Vision-language and multimodal / Multimodal LLMs
Jack of All Tasks Master of Many: Designing General-Purpose Coarse-to-Fine Vision-Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VistaLLM, a general-purpose LLM-based vision model covering coarse and fine-grained VL tasks over single and multiple images, with an instruction-guided QFormer tokenizer, gradient-aware adaptive contour sampling for masks, the CoinIt 6.8M-sample instruction dataset, and the AttCoSeg multi-image co-segmentation task.
Problem
Existing general-purpose vision-language models cannot jointly handle segmentation, multi-image inputs, and coarse-level tasks within one instruction-following framework, and uniform contour sampling loses detail when serializing masks as sequences.
Approach
Combines an instruction-guided QFormer image tokenizer that compresses global embeddings of multiple images according to the task instruction, with a curvature-based adaptive contour sampling that keeps points at sharp bends when converting masks to token sequences, trained on a large coarse-to-fine instruction dataset including a new multi-image co-segmentation task.
Key results
- VistaLLM-13B reaches 128.4 CIDEr on COCO captioning, a 10.9-point gain over Shikra-13B.
- On GRES, VistaLLM-13B reaches 65.1 gIoU, 6.7 above LAVT, and on GREC 54.6 precision, 13.1 above MDETR.
- On iCoSeg, VistaLLM-13B reaches 95.1 average Jaccard, 3.0 above CycleSegNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adaptive contour sampling (gradient-aware) instead of uniform sampling for mask sequences (ablation) | segmentation as sequence (RES) · RefCOCO val, upper bound with 32 points | mIoU upper bound: 97.26 vs 94.70 | Helped |
| increase sampled points from 16 to 32 for adaptive sampling (ablation) | VistaLLM RES · Ref and Ref+ val | mIoU: +3.6 on Ref, +4.5 on Ref+ | Helped |
| remove instruction-guided image tokenizer (ablation) | VistaLLM-13B · iCoSeg | Av. J-index: -5.4 (95.1 to 89.7) | Hurt |
| remove instruction-guided image tokenizer (ablation) | VistaLLM-13B · NLVR dev | accuracy: -3.5 (80.8 to 77.3) | Hurt |
| QFormer tokenizer without pre-trained weights (ablation) | VistaLLM-13B tokenizer · iCoSeg and NLVR | Av. J / accuracy: iCoSeg 94.8 vs 95.1 (small); NLVR 79.5 vs 80.8 | Hurt |
| scale LLM from Vicuna-7B to Vicuna-13B | VistaLLM · RefCOCO RES val | mIoU: 74.5 (7B) to 77.2 (13B) | Helped |
| instruction-guided multi-image tokenizer and adaptive sampling, full VistaLLM vs prior general-purpose models | VistaLLM-13B · CoSeg iCoSeg and PASCAL | Av. J: +3.0 on iCoSeg, +3.6 on PASCAL over CycleSegNet | Helped |
| VistaLLM vs Shikra on captioning and VQA | VistaLLM-13B · COCO captioning CIDEr and VQAv2 test-dev | CIDEr / accuracy: +10.9 CIDEr, +1.5 VQAv2 test-dev | Helped |
| VistaLLM vs Ferret on REC and GRES/GREC grounding | VistaLLM-13B · RefCOCO REC val, gRefCOCO GREC/GRES | Pr@0.5 / gIoU: REC +0.4 val; GREC +13.1 over MDETR; GRES +6.7 over LAVT | Helped |
| VistaLLM vs PolyFormer on RES (specialist) | VistaLLM-13B · RefCOCO/RefCOCO+/RefCOCOg RES | mIoU: Ref+ testA -0.1 vs PolyFormer, others gain 0.4-2.0 | Mixed |
Surprising findings
- Without the pretrained QFormer weights, performance drops on most tasks, while iCoSeg is largely unaffected.
Limitations
- Weaker than GPT-4V on general and knowledge-based question answering, attributed to limited pretraining scale.
- Mask sampling with a fixed number of points imposes an upper bound on segmentation accuracy (e.g., 97.26 mIoU at 32 points on RefCOCO val).
Experimental setup
Datasets: COCO Flickr30K Entities VCR LLaVA-Instruct-150K Visual Genome PASCAL RefCOCO RefCOCO+ RefCOCOg gRefCOCO LookTwice-QA Visual-7W
Benchmarks: VQAv2 COCO Captioning RefCOCO RefCOCO+ RefCOCOg gRefCOCO LookTwice-QA Visual-7W VCR TextVQA IconQA HM
Metrics: accuracy CIDEr Precision@0.5 mIoU gIoU N-acc Jaccard index F1
Baselines: Shikra-13B Ferret-13B MiniGPT-v2 GPT4RoI-7B CycleSegNet MDETR PolyFormer LAVT
Builds on: LLaVA Shikra InstructBLIP Vicuna
Closest papers by meaning
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.89)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.89)
- VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions (CVPR 2026 · similarity 0.88)
- Osprey: Pixel Understanding with Visual Instruction Tuning (CVPR 2024 · similarity 0.88)
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.