Research map / Vision-language and multimodal / Human and embodied modeling
Instruction-guided Multi-Granularity Segmentation and Captioning with Large Multimodal Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MGLMM, an LMM (LLaVA-style, Vicuna-7B + CLIP + SAM decoder) that switches between panoptic and fine-grained segmentation-and-captioning, with a new MGSCData benchmark (10K SAM images) and a unified SegCap data format.
Problem
Existing instruction-based LMMs produce segmentation masks and text only at the instance level, and cannot perform aligned mask-text responses across granularities from panoptic to fine-grained parts.
Approach
Combine an LLaVA-style LMM with a SAM mask decoder via [SEG] embeddings, add <p>
Key results
- MGLMM reaches 104.6 CIDEr on Flickr30k vs. 95.3 for GLaMM, and 50.1 CIDEr / 31.7 AP50 on GCG vs. 47.2 / 30.8 for GLaMM.
- On refCOCO+ val, MGLMM-7B scores 73.9 cIoU vs. 72.6 for GLaMM; on refCOCOg val it scores 77.2 vs. 74.2.
- On MGSC, MGLMM achieves 11.6 CIDEr and 7.4 AP50, versus 8.7 and 5.4 for fine-tuned GLaMM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add USCDF to MGLMM-7B without GranD (ablation) | MGLMM-7B · refCOCO+ val | cIoU: 67.2 -> 69.9 | Helped |
| Add USCDF to MGLMM-7B with GranD (ablation) | MGLMM-7B · refCOCO+ val | cIoU: 71.4 -> 73.2 | Helped |
| Add GranD dataset to MGLMM-7B with USCDF (ablation) | MGLMM-7B · refCOCO+ val | cIoU: 69.9 -> 73.2 | Helped |
| Add GranD dataset to MGLMM-7B with USCDF (ablation) | MGLMM-7B · GCG | CIDEr: 46.3 -> 50.1 | Helped |
| Use Llama2-13B instead of Vicuna-7B (ablation) | MGLMM with USCDF and GranD · refCOCO+ val | cIoU: 73.2 -> 73.4 | Helped |
| Use MGLMM instead of GLaMM | image-level captioning · NoCap | SPICE: 15.8 -> 15.2 | Hurt |
Surprising findings
- Using the unified SegCap format without GranD only slightly changes GCG CIDEr (46.5 to 46.3), while it improves refCOCO+ val by about 2.7 cIoU.
Limitations
- Only about 4M of the 11M GranD samples were publicly available, so pre-training used a subset.
- Multi-granularity benchmark annotations are generated automatically by GPT-4o and Qwen-VL without reported human verification in the text.
Experimental setup
Datasets: ADE20K COCO-Stuff Mapillary Vistas PACO-LVIS PASCAL-Part RefCLEF RefCOCO series COCO Caption LLaVA-150k GranDf GranD MGSCData
Benchmarks: MGSCData GCG refCOCO refCOCO+ refCOCOg ReasonSeg gRefCOCO Flickr30k NoCap
Metrics: METEOR CIDEr SPICE AP50 mIoU Mask Recall cIoU gIoU
Baselines: LISA PixelLM GSVA Osprey LaSagnA PSALM OMG-LLaVA GLaMM Kosmos-2 LEMON CoCa BLIP-2
Builds on: LISA GLaMM LLaVA SAM
Closest papers by meaning
- Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMs (CVPR 2026 · similarity 0.88)
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.88)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.88)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.87)
- FG-CLIP: Fine-Grained Visual and Textual Alignment (ICML 2025 · similarity 0.87)
- PARTONOMY: Large Multimodal Models with Part-Level Visual Understanding (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.