Research map / Vision-language and multimodal / Multimodal LLMs

Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Ref-Adv, a referring expression comprehension benchmark with hard same-category distractors and concise but non-trivial expressions, and shows that evaluated MLLMs drop sharply on it compared with RefCOCO(+/g).

Problem

Standard REC benchmarks (RefCOCO, RefCOCO+, RefCOCOg) have short expressions, few distractors, and redundant descriptors that allow shortcut grounding, so they weakly test visual reasoning of MLLMs, which now score above 90% on them.

Approach

Build REC expressions by first eliciting discriminative attributes between a hard distractor and the target (via GPT-4o on images with at least 3 same-category instances), then composing minimally sufficient expressions, including negation, followed by three-annotator human verification.

Key results

• Qwen3.5-397B-A17B-FP8 reaches 68.0% Acc@0.5 on Ref-Adv-s, and Qwen3-VL-235B-Thinking reaches 67.1%, while top models score above 90% on RefCOCO(+/g) (per the paper's claim).
• With fixed 'the one' prompt, Qwen2.5-VL-72B gets 35.1% on RefCOCO versus 21.4% on Ref-Adv, indicating less exploitable dataset bias on Ref-Adv.
• Bag-of-words ablation drops Qwen2.5-VL-72B accuracy by 16.8 points on Ref-Adv versus 9.9 points on RefCOCO, showing stronger textual reasoning demand.

Surprising findings

• Chain-of-thought gives limited benefit on RefCOCO(+/g) and can hurt, while it helps on Ref-Adv.
• Models often pick the hard distractor as the answer.

Limitations

• Full Ref-Adv is only partially released; public release is the 1,142-case Ref-Adv-s subset.
• Data generation relies on GPT-4o, and only 18.7% of LLM-authored expressions pass verification.
• Many evaluated models are closed or large, and results depend on prompt and output conventions per model.

Experimental setup

Datasets: COCO OpenImages v7 RefCOCO RefCOCO+ RefCOCOg

Benchmarks: Ref-Adv Ref-Adv-s RefCOCO RefCOCO+ RefCOCOg

Metrics: Acc@0.5 Acc@0.75 Acc@0.9 mAcc

Baselines: RefCOCO RefCOCO+ RefCOCOg GPT-4o Claude-3.5 Sonnet Gemini 2.5-Flash Gemini 2.5-Pro InternVL-3 Qwen2.5-VL Qwen3-VL Qwen3.5 GLM-4.5V

Builds on: Set-of-Marks RefCOCO Cops-Ref FineCops-Ref

Closest papers by meaning

• RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.92)
• A Large-Scale Human-Centric Benchmark for Referring Expression Comprehension in the LMM Era (NeurIPS 2024 · similarity 0.92)
• ScanFormer: Referring Expression Comprehension by Iteratively Scanning (CVPR 2024 · similarity 0.91)
• Look-Back: Implicit Visual Re-focusing in MLLM Reasoning (AAAI 2026 · similarity 0.90)
• When to Think and When to Look: Uncertainty-Guided Lookback (CVPR 2026 · similarity 0.90)
• LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension (ICLR 2025 · similarity 0.90)