Research map / Vision-language and multimodal / Multimodal LLMs
MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds the MMR dataset (194K GPT-4V-generated multi-target, object- and part-level reasoning segmentation QA pairs over PACO-LVIS) and proposes M²SA, a LISA extension with early local feature fusion and multiple [SEG] tokens.
Problem
Existing reasoning segmentation datasets (ReasonSeg, MUSE) focus on single-target or object-level reasoning and do not jointly cover multiple targets and part-level targets.
Approach
Generate implicit multi-target, multi-granularity question-answer pairs from PACO-LVIS object/part masks using GPT-4V with a two-step caption-then-QA prompt plus human filtering, and extend LISA with multiple [SEG] tokens (one per target) and early-layer SAM features fused into the mask decoder input.
Key results
- On the MMR test Obj & Part set, M²SA-7B reaches 30.9 gIoU / 46.8 cIoU versus 19.5 / 27.1 for LISA-7B retrained on MMR (LISA-7B_tr).
- M²SA-Llama2-13B achieves 31.6 gIoU / 47.6 cIoU on MMR test Obj & Part.
- On the multi-target Multi-RefCOCO benchmark, M²SA-7B achieves 71.3 cIoU on val versus 34.0 for LISA-7B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add multiple [SEG] tokens (ablation) | M²SA-7B · MMR test Part | gIoU: 8.0 -> 11.7 | Helped |
| add early local feature fusion to the multiple-[SEG]-token model (ablation) | M²SA-7B · MMR test Part | gIoU: 11.7 -> 13.5 | Helped |
| add multiple [SEG] tokens and early local feature fusion (ablation) | M²SA-7B · MMR test Obj & Part | gIoU: 19.4 -> 31.0 | Helped |
| train LISA-7B on MMR instead of ReasonSeg | LISA-7B · ReasonSeg val | gIoU: 44.4 -> 49.9 | Helped |
| use M²SA-7B instead of GLaMM on single-target RefCOCO | RefCOCO · RefCOCO val | cIoU: 79.5 -> 74.0 | Hurt |
| use M²SA-7B instead of GSVA-7B for multi-target segmentation | Multi-RefCOCO · Multi-RefCOCO val | cIoU: 50.7 -> 71.3 | Helped |
| use M²SA-7B instead of GSVA-7B on unseen driving-scene data | multi-object segmentation · GTAV-val | mIoU: 15.7 -> 35.1 | Helped |
Surprising findings
- Models trained on MMR outperform the ReasonSeg-trained LISA-7B on the ReasonSeg validation set (cIoU 55.6 vs 46.0), suggesting the dataset transfers to other reasoning segmentation data.
- Adding multiple [SEG] tokens alone yields large gains, indicating single [SEG] token designs are inadequate for multi-target segmentation.
Limitations
- PACO-LVIS lacks human class and part annotations, so person-related reasoning is not covered.
- QA content depends on the pre-trained knowledge of the GPT-4V generator.
- Performance on infrequent object categories drops notably (cIoU 39.7 on the lowest-frequency object subset).
Experimental setup
Datasets: MMR PACO-LVIS ADE20K COCO-Stuff Mapillary PASCAL-Part RefCOCO RefCOCO+ RefCOCOg RefCLEF LLaVA-Instruct-150K ReasonSeg
Benchmarks: MMR RefCOCO RefCOCO+ RefCOCOg Multi-RefCOCO RefCOCOm ReasonSeg GTAV Cityscapes
Metrics: gIoU cIoU mIoU
Baselines: LISA GSVA GLaMM PixelLM MCN VLT CRIS LAVT ReLA SeqTR UniRES
Builds on: LISA LLaVA SAM PACO-LVIS GSVA
Closest papers by meaning
- MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning (ICLR 2026 · similarity 0.89)
- LISA: Reasoning Segmentation via Large Language Model (CVPR 2024 · similarity 0.88)
- MMR-V: What's Left Unsaid? A Benchmark for Multimodal Deep Reasoning in Videos (ICLR 2026 · similarity 0.88)
- DR$^2$Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models (ICML 2026 · similarity 0.87)
- MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.87)
- MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.