Research map / Vision-language and multimodal / Human and embodied modeling
GLaMM: Pixel Grounding Large Multimodal Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces GLaMM, an end-to-end large multimodal model that interleaves natural-language responses with pixel-level segmentation masks and accepts text and region prompts, plus a new GCG task and the GranD/GranDf datasets.
Problem
Existing large multimodal models either produce ungrounded text or ground only single objects, require user-specified regions, or lack dense pixel-level grounding in conversations.
Approach
Combine a global CLIP image encoder, a region encoder (RoIAlign over CLIP features), a Vicuna LLM, and a SAM-based grounding encoder and pixel decoder, with <SEG> tokens so the LLM emits grounded phrases with masks in one end-to-end pass; pair with a large automatically annotated dataset (GranD) and a new Grounded Conversation Generation task.
Key results
- On the GCG task GLaMM reaches 43.5 CIDEr, 29.2 AP50 and 40.8 mask recall on the test split, ahead of LISA* (32.2 CIDEr, 24.8 AP50, 35.5 recall).
- On referring expression segmentation GLaMM reaches 79.5/83.2/76.9 on refCOCO val/testA/testB, above LISA-7B (74.9/79.1/72.3).
- Zero-shot image captioning on Flickr30k reaches 95.3 CIDEr and 18.8 SPICE, above Kosmos-2 (66.7 CIDEr).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GLaMM instead of the adapted LISA baseline | GCG · GCG test | METEOR: 12.9 -> 15.8 | Helped |
| Use GLaMM instead of Kosmos-2 | GCG · GCG test | CIDEr: 27.2 -> 43.5 | Helped |
| Use GLaMM instead of the adapted LISA baseline | GCG · GCG test | AP50: 24.8 -> 29.2 | Helped |
| Use GLaMM instead of the adapted LISA baseline | GCG · GCG test | mIoU: 61.7 -> 65.6 | Helped |
| Use GLaMM instead of the adapted LISA baseline | GCG · GCG test | Mask Recall: 35.5 -> 40.8 | Helped |
| Exclude the 1K human-annotated images from GranDf training (ablation) | GLaMM · GCG test | METEOR: 15.8 -> 14.6 | Hurt |
| Exclude the 1K human-annotated images from GranDf training (ablation) | GLaMM · GCG test | CIDEr: 43.5 -> 37.9 | Hurt |
| Exclude the 1K human-annotated images from GranDf training (ablation) | GLaMM · GCG test | AP50: 29.2 -> 27.2 | Hurt |
| Exclude the 1K human-annotated images from GranDf training (ablation) | GLaMM · GCG test | mIoU: 65.6 -> 64.6 | Hurt |
| Exclude the 1K human-annotated images from GranDf training (ablation) | GLaMM · GCG test | Mask Recall: 40.8 -> 38.0 | Hurt |
| Use GLaMM instead of LISA-7B | referring-expression segmentation · refCOCOg val(U) | IoU: 67.9 -> 74.2 | Helped |
| Use GLaMM instead of LISA-7B | referring-expression segmentation · refCOCOg test(U) | IoU: 70.6 -> 74.9 | Helped |
Limitations
- Evaluation on GCG relies on a newly proposed benchmark built from GPT-4-revised data and a small manual subset.
- Region-level grounding quality depends on automatic annotation (GranD) with verification limited to checks rather than full human review.
Experimental setup
Datasets: GranD GranDf Flickr-30K RefCOCOg PSG COCO captions refCOCO refCOCO+ Visual Genome
Benchmarks: GranDf GCG validation/test refCOCO refCOCO+ refCOCOg Visual Genome Flickr30k NoCap
Metrics: METEOR CIDEr AP50 mIoU mask recall SPICE cIoU
Baselines: BuboGPT Kosmos-2 LISA GRiT GPT4RoI CRIS LAVT GRES
Builds on: LLaVA SAM LISA GPT4RoI CLIP
Closest papers by meaning
- VideoGLaMM : A Large Multimodal Model for Pixel-Level Visual Grounding in Videos (CVPR 2025 · similarity 0.92)
- F-LMM: Grounding Frozen Large Multimodal Models (CVPR 2025 · similarity 0.92)
- GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models (AAAI 2026 · similarity 0.90)
- Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding (CVPR 2025 · similarity 0.89)
- Grounding Multimodal Large Language Models to the World (ICLR 2024 · similarity 0.89)
- GroundingFace: Fine-grained Face Understanding via Pixel Grounding Multimodal Large Language Model (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.