Research map / Detection and segmentation / Open-set and few-shot detection
ROD-MLLM: Towards More Reliable Object Detection in Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ROD-MLLM, an MLLM for language-based object detection that uses an open-vocabulary detector as a query-based low-level locator plus an LLM for high-level comprehension, trained on a new automatically annotated ROD dataset (~500K description-image pairs).
Problem
Existing MLLMs only ground objects that exist in the image and cannot reliably reject non-existent objects or handle multiple matching objects for free-form language descriptions.
Approach
Decouple language-based detection into a low-level OVD locator that proposes candidate boxes from query nouns and a Vicuna LLM that receives global and region tokens (2x2 patches per region plus anchor tokens) to decide which candidates match or return None; training data is built automatically from detection and grounding datasets using MLLM-generated descriptions, condition decomposition, and ch
Key results
- On OmniLabel, ROD-MLLM (7B) reaches AP-d 25.3 vs 15.6 for the best generalist MLLM Groma, a gain of +9.7.
- On D3 Full, ROD-MLLM achieves 29.7 AP vs 16.0 for MLLMs, a 86% relative improvement (+13.7).
- On gRefCOCO testB, ROD-MLLM improves Pr by 20.7% over the best specialist-level baseline reported in the table; on RefCOCOg region captioning it improves CIDEr by 6.5 over Groma-level SOTA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ROD-MLLM instead of FIBER-B | language-based detection · OmniLabel | AP-d: 22.3 -> 25.3 | Helped |
| use ROD-MLLM instead of InternVL2-76B | language-based detection · D3 | Full AP: 25.3 to 29.7 | Helped |
| use ROD-MLLM instead of Groma | referring expression comprehension · RefCOCO+ val | Acc@0.5: 83.9 -> 84.8 | Helped |
| use ROD-MLLM instead of UNINEXT-R50 | generalized referring expression comprehension · gRefCOCO test-A | N-acc: 49.3 -> 63.7 | Helped |
| use ROD-MLLM instead of Groma | region captioning · RefCOCOg | CIDEr: 107.3 -> 113.8 | Helped |
| replace OWLv2 with Grounding DINO-T as the low-level locator (ablation) | ROD-MLLM · OmniLabel | AP-d: 25.3 -> 25.9 | No clear effect |
| use only common COCO objects instead of query-based candidate boxes (ablation) | ROD-MLLM · OmniLabel | AP-d: 25.3 -> 10.8 | Hurt |
| use only query-extracted objects as candidate boxes (ablation) | ROD-MLLM · OmniLabel | AP-d: 25.3 -> 24.4 | No clear effect |
| add ROD detection data to training (ablation) | ROD-MLLM · OmniLabel | AP-d: 18.8 -> 23.1 | Helped |
| add ROD grounding data to training (ablation) | ROD-MLLM · OmniLabel | AP-d: 18.8 -> 23.2 | Helped |
| add mixed ROD detection and grounding data to training (ablation) | ROD-MLLM · OmniLabel | AP-d: 18.8 -> 24.1 | Helped |
Surprising findings
- Swapping OWLv2 for Grounding DINO-T as the low-level locator changes final metrics by no more than about 1 AP, suggesting the LLM rather than the locator drives performance.
- Using only COCO common categories as candidate boxes drops OmniLabel AP-d to 10.8, far below query-based box extraction (24.4).
Limitations
- Relies on an external open-vocabulary detector for candidate boxes, so recall is bounded by the locator's proposals.
- The ROD annotations are generated by MLLMs (InternVL2-76B, DeepSeek) and have no human verification described, so label noise is possible.
Experimental setup
Datasets: ROD (constructed) Objects365 Flickr30K Entities COCO RefCOCO/+/g gRefCOCO GRIT-20M Visual Genome LAION-CCSBU LLaVA Instruct 665K VCR
Benchmarks: OmniLabel D3 RefCOCO/+/g gRefCOCO RefCOCOg Visual Genome
Metrics: AP-d AP-dP AP-dS/M/L AP (Full/Presence/Absence) Acc@0.5 Pr@(F1=1, IoU>=0.5) N-acc METEOR CIDEr
Baselines: OWLv2 GLIP-T FIBER-B Grounding DINO InternVL2-76B InternVL2-8B Griffon Groma Ferret Shikra Pink MDETR
Builds on: Groma Shikra OWLv2 LLaVA Objects365
Closest papers by meaning
- Roboflow100-VL: A Multi-Domain Object Detection Benchmark for Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors (ICLR 2024 · similarity 0.89)
- Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction (CVPR 2025 · similarity 0.88)
- MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs (ICLR 2025 · similarity 0.88)
- On Scaling Up a Multilingual Vision and Language Model (CVPR 2024 · similarity 0.88)
- Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.