Research map / Vision-language and multimodal / Human and embodied modeling
A Large-Scale Human-Centric Benchmark for Referring Expression Comprehension in the LMM Era
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces HC-RefLoCo, a large human-centric referring expression comprehension benchmark with long GPT-4V-generated, manually reviewed descriptions, and evaluates 24 REC models under multiple IoU, scale, and subject protocols.
Problem
Existing human-centric referring expression comprehension (REC) benchmarks derived from RefCOCO-family datasets are small and use very short expressions, which is inadequate for evaluating modern large multimodal models.
Approach
Build a large benchmark of human instances from detection datasets and celebrity images, with multi-sentence, ~93-word descriptions generated by GPT-4V and manually reviewed, sentence-level subject labels, and richer evaluation protocols (multi-IoU mAcc, scale-aware, per-subject).
Key results
- SPHINX-v2-1k achieves the highest mAcc on the Val+Test split at 71.7, with SPHINX-MoE-1k close at 71.4.
- GPT-4V scores only 5.5 mAcc, far below open-source specialist models such as OFA-Large (58.1) and SPHINX variants.
- Most models degrade as instance size decreases; CogVLM shows the greatest robustness across scales.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use all annotation sentences instead of only sentences for the selected subject (ablation) | SPHINX · HC-RefLoCo subject-specific evaluation | mAcc | Helped |
| Use all annotation sentences instead of only sentences for the selected subject (ablation) | Shikra · HC-RefLoCo subject-specific evaluation | mAcc | Helped |
| Use contextual descriptions in the annotations (ablation) | MiniGPT4 v2 · HC-RefLoCo subject-specific evaluation | mAcc | Hurt |
| Reduce instance size | Most evaluated models · HC-RefLoCo instance-size groups | mAcc | Hurt |
| Use different numbers of randomly selected annotation sentences (1, 3, 5, or all) (ablation) | Evaluated REC models · HC-RefLoCo sentence-count sets | mAcc | Mixed |
Surprising findings
- GPT-4V performs very poorly (5.5 mAcc) on the grounding task despite strong general vision-language ability.
- MiniGPT4-v2 declines sharply when annotations include more contextual sentences, showing difficulty associating long text with visual elements.
Limitations
- Benchmark limited to six subjects and 13,452 images; test set size could be larger.
- Some unintentional annotation errors may remain despite manual review.
- Annotations are GPT-4V-generated and may contain hallucinations despite manual correction.
Experimental setup
Datasets: COCO 2017 Objects365 OpenImage v7 LAION-5B
Benchmarks: HC-RefLoCo
Metrics: Acc0.5 Acc0.75 Acc0.9 mAcc scale-aware mAcc (small/medium/large) per-subject mAcc
Baselines: GPT-4V GroundingGPT Ferret MiniGPT-4 v2 KOSMOS-2 Shikra OFA Qwen-VL CogVLM Lenna ONE PEACE SPHINX
Builds on: RefCOCO RefCOCO+ RefCOCOg SPHINX LISA
Closest papers by meaning
- Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Tasks (ICLR 2026 · similarity 0.92)
- RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression Comprehension (AAAI 2025 · similarity 0.89)
- Human-MME: A Holistic Evaluation Benchmark for Human-Centric Multimodal Large Language Models (ICLR 2026 · similarity 0.88)
- Rethinking Two-Stage Referring Expression Comprehension: A Novel Grounding and Segmentation Method Modulated by Point (AAAI 2024 · similarity 0.88)
- ScanFormer: Referring Expression Comprehension by Iteratively Scanning (CVPR 2024 · similarity 0.88)
- MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.