Research map / Vision-language and multimodal / Multimodal LLMs
CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoVLM, which inserts communication tokens into a Pythia LLM so it alternates between a detection network (YOLOX-style over CLIP features) and language generation, improving zero-shot compositional reasoning on ARO, Cola, and HICO-DET.
Problem
Current vision-language models behave like bag-of-words and fail at compositional reasoning over visual entities and their relations, since they encode the whole image holistically and do not link generated words to specific regions.
Approach
Special communication tokens (<obj>, <visual>, <box>, <previsual>, <prebox>) let the LLM alternate top-down, querying a detection network with its hidden state to propose ROIs, and bottom-up, feeding pooled ROI features back into the token stream, so each entity or relation step is grounded in image regions during decoding.
Key results
- On HICO-DET, CoVLM 1.4B reaches 39.00 full mAP and 50.82 rare mAP zero-shot, versus 33.51 rare / 21.26 full for KOSMOS-2.
- On Cola multi-object, CoVLM 1.4B reaches 44.29 top-1 accuracy, above BLIP at 41.43 and the supervised CLIP+MM-Pred at 41.42.
- On ARO, CoVLM 1.4B reaches 32.46 top-1 and 55.70 top-5 accuracy, versus 29.78 / 54.18 for BLIP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CoVLM's communicative decoding instead of the strongest listed baseline | CoVLM 1.4B vs. BLIP · ARO | Top-1 accuracy: 29.78 -> 32.46 | Helped |
| Use CoVLM's communicative decoding instead of the strongest listed baseline | CoVLM 1.4B vs. BLIP-2 ViT-L OPT 2.7B · ARO | Top-5 accuracy: 54.91 -> 55.70 | Helped |
| Use CoVLM's communicative decoding instead of BLIP | CoVLM 1.4B · Cola | Top-1 accuracy: 41.43 -> 44.29 | Helped |
| Use CoVLM's communicative decoding instead of KOSMOS-2 | CoVLM 1.4B · HICO-DET Rare | mAP: 33.51 -> 50.82 | Helped |
| Use CoVLM's communicative decoding instead of KOSMOS-2 | CoVLM 1.4B · HICO-DET Non-Rare | mAP: 17.83 -> 35.47 | Helped |
| Use CoVLM's communicative decoding instead of KOSMOS-2 | CoVLM 1.4B · HICO-DET Full | mAP: 21.26 -> 39.00 | Helped |
| Use zero-shot CoVLM 1.4B instead of task-specific supervised CLIP + MM-Pred | CoVLM 1.4B · Cola | accuracy: 41.42 -> 44.29 | Helped |
| Use zero-shot CoVLM instead of supervised RLIPv2-ParSeDA | CoVLM 1.4B · HICO-DET Rare | mAP: 43.23 -> 50.82 | Helped |
| Use zero-shot CoVLM instead of supervised RLIPv2-ParSeDA | CoVLM 1.4B · HICO-DET Non-Rare | mAP: 45.64 -> 35.47 | Hurt |
| Use zero-shot CoVLM instead of supervised RLIPv2-ParSeDA | CoVLM 1.4B · HICO-DET Full | mAP: 45.09 -> 39.00 | Hurt |
| Scale the language backbone from Pythia-1.4B to Pythia-2.8B | CoVLM · RefCOCOg val | accuracy: 60.87 -> 61.23 | Helped |
| Scale the language backbone from Pythia-1.4B to Pythia-2.8B | CoVLM · RefCOCO+ val | accuracy: 47.62 -> 48.87 | Helped |
Surprising findings
- Zero-shot CoVLM exceeds all supervised HICO-DET methods on the Rare split (50.82 vs 43.23 for RLIPv2-ParSeDA).
- Alignment models (CLIP, FLAVA) score lower on ARO than generative models, which the authors attribute to bag-of-words behavior from contrastive training.
Limitations
- Object-attribute compositionality and spatial event compositionality are not addressed, per the authors.
- On RefCOCO, CoVLM trails KOSMOS-2 (48.19 vs 52.32 val), and VQAv2 results show a margin versus BLIP-2 and Flamingo.
- Evaluation relies on perplexity-based selection and manually built prompts, and the VQAv2 human evaluation used only 1000 samples.
Experimental setup
Datasets: BLIP-2 pre-training image-text data (COCO, CC3M, CC12M, Visual Genome, SBU, LAION subset) The Pile ARO Cola HICO-DET RefCOCO RefCOCO+ RefCOCOg VQAv2
Benchmarks: ARO Cola HICO-DET RefCOCO RefCOCO+ RefCOCOg VQAv2
Metrics: top-1 accuracy top-5 accuracy mAP perplexity-based selection accuracy human-evaluated VQA accuracy
Baselines: CLIP FLAVA OpenFlamingo BLIP BLIP-2 KOSMOS-2 RLIPv2-ParSeDA CDN
Builds on: KOSMOS-2 Pythia YOLOX CLIP BLIP-2
Closest papers by meaning
- VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks (NeurIPS 2024 · similarity 0.90)
- ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs (NeurIPS 2024 · similarity 0.90)
- Not Only Text: Exploring Compositionality of Visual Representations in Vision-Language Models (CVPR 2025 · similarity 0.90)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
- Causal Graphical Models for Vision-Language Compositional Understanding (ICLR 2025 · similarity 0.90)
- Natural Language Inference Improves Compositionality in Vision-Language Models (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.