Research map / Vision-language and multimodal / Multimodal LLMs
Unleashing Region Understanding in Intermediate Layers for MLLM-based Referring Expression Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free "unleash-then-eliminate" decoding framework that contrasts intermediate and final layers of an MLLM to generate richer referring expressions, then uses a RES model's cycle-consistency IoU to rank candidates and probing-based layer importance weights to reduce RES calls.
Problem
MLLM-based referring expression generation faces a trade-off: more detailed descriptions increase hallucinations, so generating precise, unambiguous region descriptions without hallucination is difficult.
Approach
Elicit region-descriptive information from intermediate layers via contrastive decoding against the final layer, rank candidate captions by cycle consistency using a RES model (IoU of the regenerated mask with the input mask), and estimate per-layer prior importance on a small probing subset so the RES model can be dropped at inference time.
Key results
- On RefCOCOg, the full-R method reaches METEOR 173.0 and nCHAIRS 1.831 versus 162.0 / 3.274 for Osprey-7b (t=0.2) and 168.0 / 1.883 for DoLa.
- The probing-based estimate from a 1/8 subset (METEOR 172.0, nCHAIRI 1.348) performs comparably to full-dataset ranking while avoiding per-sample RES scoring.
- On PHD (neutral mode), the method averages 70.88% versus 69.91% for Osprey-7b (t=0.2) and 70.27% for DoLa.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply the proposed 1/8 probing-based layer importance decoding instead of DoLa | Osprey-7b · RefCOCOg | METEOR: 168.0 -> 172.0 | Helped |
| apply the proposed 1/8 probing-based layer importance decoding instead of DoLa | Osprey-7b · RefCOCOg | nCHAIRI: 1.3775 -> 1.3484 | Helped |
| apply the proposed decoding method instead of DoLa | Osprey-7b · PHD neutral | Average: 70.27 -> 70.88 | Helped |
| apply the proposed decoding method instead of DoLa | Osprey-7b · PHD misleading | Average: 67.27 -> 68.69 | Helped |
| use candidate layers [0, 7] rather than [24, 31] (ablation) | Osprey-7b with cycle-consistency ranking · RefCOCOg | METEOR: 127.0 -> 173.0 | Helped |
| use candidate layers [0, 7] rather than [16, 23] (ablation) | Osprey-7b with cycle-consistency ranking · RefCOCOg | METEOR: 146.0 -> 173.0 | Helped |
| use GLaMM instead of LISA as the RES scorer (ablation) | cycle-consistency-based ranking · RefCOCOg | METEOR: 171.0 -> 172.0 | Helped |
| increase the probing subset from 1/16 to 1/8 (ablation) | Osprey-7b layer importance estimation · RefCOCOg | METEOR: 171.0 -> 172.0 | Helped |
| use the best sampled 1/8 subset rather than average random 1/8 sampling (ablation) | Osprey-7b layer importance estimation · RefCOCOg | nCHAIRS: 1.840 -> 1.829 | Helped |
Surprising findings
- CHAIR metrics favor short captions, so the authors introduce a length-normalized nCHAIR variant to compare sentences of different lengths fairly.
- Intermediate layers (e.g., layers 0-7 group) produce more descriptive region expressions than the final layer, and semantic quality tends to decline as layer depth increases after contrastive decoding.
Limitations
- Training-free approach may underperform training-based methods tuned on the target dataset.
- Performance depends on the quality of the RES model used for cycle-consistency scoring, and adds computational overhead.
- Probing subset randomness affects results; 1/16 subset is less stable than 1/8.
Experimental setup
Datasets: RefCOCOg PHD
Benchmarks: RefCOCOg PHD
Metrics: METEOR CHAIRS CHAIRI nCHAIRS nCHAIRI Recall accuracy on PHD
Baselines: Osprey-7b (t=0.2) Osprey-7b (t=0.9) DoLa
Builds on: DoLa Osprey GLaMM CHAIR
Closest papers by meaning
- MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation (ICLR 2025 · similarity 0.90)
- When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understanding (NeurIPS 2025 · similarity 0.90)
- MoLE:Decoding by Mixture of Layer Experts Alleviates Hallucination in Large Vision-Language Models (AAAI 2025 · similarity 0.90)
- Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment (ICLR 2025 · similarity 0.90)
- Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decoding (CVPR 2025 · similarity 0.89)
- Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.