Research map / Vision-language and multimodal / Multimodal LLMs

Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Rex-Thinker reformulates object referring as explicit planning-action-summarization chain-of-thought reasoning over candidate box hints, trained with CoT SFT on the new HumanRef-CoT dataset followed by GRPO.

Problem

Object referring (REC) models predict boxes directly with limited interpretability and struggle to abstain when no object matches the expression (hallucination on rejection cases).

Approach

Treat referring as retrieval over detector-proposed candidate boxes and have an MLLM reason step by step (plan, check each candidate, summarize) inside think/answer tags, with cold-start SFT on GPT-4o-generated CoT traces followed by GRPO with an F1 reward.

Key results

• On HumanRef, Rex-Thinker-GRPO reaches average R/P/DF1 of 86.6/86.8/83.5 and rejection score 68.2, versus 85.8/85.9/82.3 average and 54.1 rejection for RexSeek-7B.
• CoT SFT raises rejection score from 53.5 (Plain) to 67.3, a 13.8-point gain.
• Cold-start SFT before GRPO improves average DF1 from 77.8 to 83.5 on HumanRef.

Surprising findings

• GPT-4o without answer hints reaches only 53.2 average DF1 on HumanRef, indicating limited standalone performance despite being used as annotator with ground-truth hints.
• Without box hints, the model over-rejects, abstaining on 189 non-rejection samples versus 134 with hints, which partly explains its higher rejection score.

Limitations

• Weaker performance on the interaction subset, where multi-object relational reasoning errors propagate.
• Inconsistencies between reasoning traces and final box outputs (e.g., nine candidates reasoned but eight output).
• Higher inference latency due to long CoT outputs; relies on an external open-vocabulary detector for candidate boxes.

Experimental setup

Datasets: HumanRef HumanRef-CoT RefCOCOg

Benchmarks: HumanRef RefCOCOg

Metrics: Recall Precision DensityF1 rejection score accuracy@IoU0.5

Baselines: DINOX InternVL-2.5-8B Ferret-7B Groma-7B ChatRex-7B Qwen2.5-VL-7B DeepSeek-VL2-small Molmo-7B-D RexSeek-7B GPT-4o-CoT

Builds on: DeepSeek-R1 GRPO Qwen2.5-VL HumanRef Set-of-Mark

Closest papers by meaning

• RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.89)
• Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks (ICLR 2026 · similarity 0.89)
• Referencing Where to Focus: Improving Visual Grounding with Referential Query (NeurIPS 2024 · similarity 0.87)
• Seeing Clearly, Reasoning Confidently: Plug-and-Play Remedies for Vision Language Model Blindness (CVPR 2026 · similarity 0.87)
• See Say and Segment: Teaching LMMs to Overcome False Premises (CVPR 2024 · similarity 0.87)
• ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension (ICLR 2025 · similarity 0.87)