Research map / Vision-language and multimodal / Multimodal LLMs
Grounding Multimodal Large Language Models to the World
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Kosmos-2 extends the Kosmos-1 MLLM with grounding and referring abilities by training on GrIT, a ~91M-image web-scale dataset of text spans linked to bounding boxes in Markdown-style hyperlinks.
Problem
Existing multimodal LLMs can only produce text outputs and refer to image regions only through detailed text descriptions, so they cannot ground language to the visual world or accept/return bounding boxes.
Approach
Represent text spans (noun phrases and referring expressions) as Markdown-style links whose targets are discretized location tokens for bounding boxes, and train a causal MLLM on a large automatically built grounded image-text corpus (GrIT) mixed with Kosmos-1 multimodal data.
Key results
- Zero-shot phrase grounding on Flickr30k Entities test R@1 of 78.7, outperforming GRILL (18.9) but below finetuned GLIP (87.1) and FIBER (87.4).
- Zero-shot referring expression comprehension on RefCOCOg test reaches 61.65 accuracy, well above GRILL's 47.5, but lags finetuned models on RefCOCO/RefCOCO+.
- Zero-shot referring expression generation on RefCOCOg reaches 60.3 CIDEr, and 62.3 with 4-shot prompting, beating finetuned SLR (59.2) but below SLR+Rerank (66.2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add grounded image-text training data and grounding capability to Kosmos-1 | Kosmos-2 · Flickr30k test | R@1: 87.4 -> 78.7 | Hurt |
| Add grounded image-text training data and grounding capability to Kosmos-1 | Kosmos-2 · RefCOCOg val | accuracy: 87.11 -> 60.57 | Hurt |
| Use few-shot demonstrations (k=2) instead of zero-shot | Kosmos-2 · RefCOCOg | CIDEr: 60.3 -> 62.2 | Helped |
| Use few-shot demonstrations (k=4) instead of zero-shot | Kosmos-2 · RefCOCOg | CIDEr: 60.3 -> 62.3 | Helped |
| Add grounded image-text training data and grounding capability to Kosmos-1 | Kosmos-2 · Flickr30k test | CIDEr: 65.2 -> 66.7 | Helped |
| Add grounded image-text training data and grounding capability to Kosmos-1 | Kosmos-2 · VQAv2 test-dev | VQA accuracy: 46.7 -> 45.6 | Hurt |
| Add grounded image-text training data and grounding capability to Kosmos-1 | Kosmos-2 · BoolQ | score: 56.4 -> 62.0 | Helped |
| Add grounded image-text training data and grounding capability to Kosmos-1 | Kosmos-2 · CB | score: 44.6 -> 30.4 | Hurt |
| Add grounded image-text training data and grounding capability to Kosmos-1 | Kosmos-2 · COPA | score: 63.0 -> 67.0 | Helped |
Surprising findings
- Kosmos-2 scores lower than Kosmos-1 on CB and VQAv2 and the paper reports small gaps on several language tasks, indicating a trade-off between new grounding skills and prior capabilities.
- Zero-shot RefCOCO/RefCOCO+ accuracy is much lower than RefCOCOg, which the authors attribute to shorter expressions in those datasets.
Limitations
- Performance on RefCOCO/RefCOCO+ trails finetuned models, attributed to short expressions and limited expression diversity.
- Grounded training labels come from an automatic pipeline using GLIP with a 0.65 confidence threshold, so noise propagates into the data.
- Slight regressions on some language and VQA tasks relative to Kosmos-1.
Experimental setup
Datasets: GrIT LAION-2B COYO-700M Flickr30k Entities RefCOCO RefCOCO+ RefCOCOg VQAv2 LLaVA-Instruct Unnatural Instructions FLANv2 StoryCloze
Benchmarks: Flickr30k Entities RefCOCO RefCOCO+ RefCOCOg RefCOCOg referring expression generation Flickr30k VQAv2 StoryCloze HellaSwag Winograd Winogrande PIQA
Metrics: R@1 R@5 R@10 accuracy METEOR CIDEr VQA accuracy
Baselines: VisualBert MDETR GLIP FIBER GRILL UNITER OFA VisionLLM SLR Flamingo FewVLM MetaLM
Builds on: Kosmos-1 MAGNETO GLIP LLaVA-Instruct FLANv2
Closest papers by meaning
- GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation (CVPR 2026 · similarity 0.91)
- Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception (CVPR 2024 · similarity 0.90)
- PostAlign: Multimodal Grounding as a Corrective Lens for MLLMs (ICLR 2026 · similarity 0.90)
- Grounding Everything in Tokens for Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- Breaking the Regional Perception Bottleneck of Multimodal Large Language Models via External Reasoning Framework (CVPR 2026 · similarity 0.90)
- GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.