Research map / Vision-language and multimodal / Multimodal LLMs
From Pixels to Logic: A Perception-Reasoning Decomposition Framework for Open-World Referring Expression Comprehension
Lihong Huang, Sheng-hua Zhong, Zhi Zhang, Yan Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a perception-reasoning decomposition framework for open-world referring expression comprehension, where expressions may refer to zero, one, or multiple objects.
Problem
Referring expression comprehension usually assumes each expression refers to exactly one known object; real-world expressions may refer to multiple objects, none, or novel categories with complex semantics.
Approach
Decomposes open-world REC into a perception stage that proposes candidate objects and a reasoning stage that resolves the expression's logic over them, allowing zero, one, or many referents.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| enable explicit reasoning (thinking mode) (ablation) | Qwen3-32B · RefCOCO testA | accuracy: 69.8 -> 78.9 | Improved |
| enable explicit reasoning (thinking mode) (ablation) | Qwen3-235B-A22B · RefCOCO testA | accuracy: 71.4 -> 78.4 | Improved |
| add image caption (ablation) | PRDF · gRefCOCO testB | precision: 42.2 -> 52.1 | Improved |
| add containment relations (ablation) | PRDF · gRefCOCO testB | precision: 45.0 -> 52.1 | Improved |
| add depth information (ablation) | PRDF · gRefCOCO testB | precision: 52.1 -> 52.1 | No clear effect |
| add all three scene representations (ablation) | PRDF · gRefCOCO testB | precision: 37.1 -> 52.1 | Improved |
| add prompt note about small-region hallucination (ablation) | PRDF · RefCOCO testA | accuracy: 80.7 -> 81.3 | Improved |
| use more generated masks (ablation) | PRDF · gRefCOCO | F1: Pearson r between -0.10 and -0.05 | Hurt |
| use reasoning LM Deepseek-R1 vs non-reasoning Deepseek-V3 (ablation) | PRDF · RefCOCO testA | accuracy: 77.6 -> 80.7 | Improved |
| compare zero-shot PRDF with supervised SimVG on no-target accuracy | SimVG · gRefCOCO val | no-target accuracy: 54.7 -> 85.2 | Improved |
| compare zero-shot PRDF with supervised SimVG on precision | SimVG · gRefCOCO | precision: 62.1 -> 65.4 (val); 54.8 -> 52.1 (testB) | Mixed |
Closest papers by meaning
- Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks (ICLR 2026 · similarity 0.93)
- ScanFormer: Referring Expression Comprehension by Iteratively Scanning (CVPR 2024 · similarity 0.90)
- ClawMachine: Learning to Fetch Visual Tokens for Referential Comprehension (ICLR 2025 · similarity 0.89)
- RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.89)
- Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.89)
- A Large-Scale Human-Centric Benchmark for Referring Expression Comprehension in the LMM Era (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
