Research map / Vision-language and multimodal / Multimodal LLMs
VCoder: Versatile Vision Encoders for Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VCoder, an adapter that feeds segmentation (and depth) maps through extra CLIP-ViT encoders into LLaVA-1.5, trained on a new COCO-based COST dataset with count, hallucination and depth scores for object perception.
Problem
Multimodal LLMs fail at basic object-level perception such as identifying, counting, and ordering objects, especially in cluttered scenes with background entities.
Approach
Add perception modalities (segmentation or depth maps) as control inputs encoded by extra vision encoders (SegCoder/DepthCoder) and projected into the LLM space, training only the MLP projectors on a COCO-derived QA dataset while keeping the base MLLM frozen.
Key results
- VCoder LLaVA-1.5-13b reaches panoptic CS 87.2 and HS 11.6 on COST val, versus GPT-4V's CS 38.4 and HS 83.0.
- VCoder LLaVA-1.5-7b improves semantic CS to 88.6 (HS 10.4) from LLaVA-1.5-7b's 30.6 (HS 60.1).
- Depth score on object order perception drops from 166.1 (LLaVA-1.5-7b) to 65.9 (VCoder-DS 7b).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add segmentation-map control input | LLaVA-1.5-7b · COST semantic | CS: 30.6 -> 88.6 | Helped |
| add segmentation-map control input | LLaVA-1.5-7b · COST instance | CS: 50.3 -> 71.1 | Helped |
| add segmentation-map control input | LLaVA-1.5-7b · COST panoptic | CS: 38.7 -> 86.0 | Helped |
| add segmentation-map and depth-map control inputs and train on COST | LLaVA-1.5-7b · COST object order | DS: 166.1 -> 65.9 | Helped |
| replace OneFormer segmentation maps with Mask R-CNN maps (ablation) | VCoder · COST instance | CS: 71.1 -> 61.9 (-9.2) | Hurt |
| replace OneFormer segmentation maps with Mask R-CNN maps (ablation) | VCoder · COST instance | HS: 26.9 -> 39.8 (+12.9) | Hurt |
| replace OneFormer segmentation maps with Mask2Former maps (ablation) | VCoder · COST panoptic | CS: 86.0 -> 76.5 (-9.5) | Hurt |
| replace OneFormer segmentation maps with Mask2Former maps (ablation) | VCoder · COST panoptic | HS: 12.8 -> 26.1 (+13.3) | Hurt |
| replace OneFormer segmentation maps with Panoptic-FPN maps (ablation) | VCoder · COST panoptic | CS: 86.0 -> 64.2 (-21.8) | Hurt |
| replace OneFormer segmentation maps with Panoptic-FPN maps (ablation) | VCoder · COST panoptic | HS: 12.8 -> 33.3 (+20.5) | Hurt |
Surprising findings
- GPT-4V performs worse than open-source LLaVA-1.5 variants on the COST object identification metrics despite strong general captioning.
- Replacing the segmentation map with an RGB image as control input (ImCoder) yields much smaller gains than segmentation maps, and a segmentation map from older Mask R-CNN drops CS notably.
Limitations
- Relies on OneFormer segmentation trained on a closed COCO vocabulary, limiting object categories.
- Count, hallucination and depth scores depend on manually defined synonym mappings.
- Inaccurate segmentation maps cause VCoder failures.
Experimental setup
Datasets: COCO COST LLaVA-1.5 instruction tuning data NYUd
Benchmarks: COST validation set
Metrics: count score (CS) hallucination score (HS) depth score (DS)
Baselines: GPT-4V MiniGPT-4 InstructBLIP LLaVA-1.5 CogVLM COST IT LLaVA-1.5 Soft-Prompted LLaVA-1.5 ImCoder LLaVA-1.5
Builds on: LLaVA-1.5 OneFormer CLIP DINOv2 ControlNet
Closest papers by meaning
- DenseFusion-1M: Merging Vision Experts for Comprehensive Multimodal Perception (NeurIPS 2024 · similarity 0.90)
- Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.90)
- Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation (NeurIPS 2025 · similarity 0.90)
- Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders (ICLR 2025 · similarity 0.90)
- MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs (ICLR 2025 · similarity 0.89)
- Visual Perception by Large Language Model’s Weights (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.