Research map / Vision-language and multimodal / Multimodal LLMs
RegionGPT: Towards Region Understanding Vision Language Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RGPT, a region-level VLM that uses mask-pooled features from a refined CLIP feature map, task-guided instruction prompts, and an automatically generated detailed region caption dataset (RecapD) to handle any-shape regions.
Problem
Vision language models struggle with detailed region-level understanding because vision encoders have limited spatial awareness and training data lacks detailed region-specific captions; prior region-aware MLLMs encode regions as text coordinates or boxes only.
Approach
Refine low-resolution CLIP features with deconvolution layers to 4x resolution, extract any-shape region embeddings via Mask Pooling and feed them into the LLM alongside image tokens, combined with task-guided output-format prompts and a two-stage GPT/LLaVA-assisted region caption pipeline.
Key results
- Achieves 70.0 mAP on COCO-2017 val region classification with ground-truth boxes, above ASM (69.3) and PVIT/GPT4RoI-based baselines (64.5/64.0 accuracy).
- Reaches 16.9 METEOR and 109.9 CIDEr on RefCOCOg region captioning, surpassing GRIT (71.6 CIDEr) and Kosmos-2 (62.3 CIDEr).
- Obtains 86.96% accuracy on ReferCOCOg test REC within provided region proposals, versus 84.66 for MiniGPT-V2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RGPT instead of ASM | COCO-2017 val region classification · COCO-2017 val | mAP: 69.3 -> 70.0 | Helped |
| Use RGPT instead of the strongest captioning baseline | RefCOCOg region captioning · RefCOCOg | CIDEr: 71.6 -> 109.9 | Helped |
| Use RGPT instead of GRIT | Visual Genome region captioning · Visual Genome | METEOR: 17.1 -> 17.0 | Hurt |
| Use RGPT instead of MiniGPT-V2 | ReferCOCOg referring expression comprehension · ReferCOCOg val and test | accuracy: 84.44 -> 86.44; 84.66 -> 86.96 | Helped |
| Use RGPT instead of InstructBLIP | POPE Random · POPE Random | accuracy: 88.57 -> 87.80 | Hurt |
| Use RGPT instead of Shikra | POPE Popular and Adversarial · POPE Popular and Adversarial | accuracy: 83.97 -> 87.20; 83.10 -> 85.67 | Helped |
| Add two deconvolution layers for feature refinement (ablation) | RGPT region classification · COCO-2017 val | AP: 57.7 -> 66.8 | Helped |
| Use bilinear upsampling instead of two deconvolution layers (ablation) | RGPT region classification · COCO-2017 val | AP: 66.8 -> 60.9 | Hurt |
| Add bilinear upsampling to the deconvolution output (ablation) | RGPT region classification · COCO-2017 val | AP: 66.8 -> 62.7 | Hurt |
| Replace the visual backbone with SigLip ViT-SO400M-384 (ablation) | RGPT region classification · COCO-2017 val | AP: 70.0 -> 71.0 | Helped |
| Classify RoIs sequentially across multiple turns instead of all in one turn (ablation) | RGPT region classification · COCO-2017 val | mAP: 70.0 -> 73.8 | Helped |
| Increase training annotations per category (ablation) | RGPT region classification · COCO-2017 val | mAP: 52.7 -> 56.8 -> 57.3 -> 60.9 | Helped |
Surprising findings
- Multi-turn classification (one RoI per turn, with prior predictions as context) outperforms one-turn-for-all-RoIs, reaching 73.8 mAP vs 70.0.
- Feature refinement makes the model robust to dilated/eroded masks, with dilated masks even improving mAP (71.0 vs 70.0).
Limitations
- Task-guided prompts do not always restrict outputs to the candidate label set; synonyms or out-of-set labels occur.
- REC evaluated with provided candidate boxes rather than true localization, limiting comparability.
- Captions for RecapD are generated by LLaVA/GPT-4 with no thorough quality audit beyond CLIP similarity filtering.
Experimental setup
Datasets: LAION-CC-SBU-558K Visual Genome ReferCOCOg V3Det COCO-2017 LVIS RefCOCO RecapD
Benchmarks: COCO-2017 val RefCOCOg Visual Genome POPE
Metrics: mAP accuracy METEOR CIDEr precision recall F1
Baselines: CLIP RegionCLIP LLaVA Shikra GPT4RoI PVIT ASM GRIT SLR Kosmos-2 MDETR MiniGPT-V2
Builds on: LLaVA CLIP Vicuna GPT4RoI RegionCLIP
Closest papers by meaning
- SpatialRGPT: Grounded Spatial Reasoning in Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- Omni-RGPT: Unifying Image and Video Region-level Understanding via Token Marks (CVPR 2025 · similarity 0.91)
- Locality Alignment Improves Vision-Language Models (ICLR 2025 · similarity 0.90)
- RegionReasoner: Region-Grounded Multi-Round Visual Reasoning (ICLR 2026 · similarity 0.89)
- GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models (ICLR 2026 · similarity 0.89)
- VLM-R³: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.