Research map / Vision-language and multimodal / Multimodal LLMs
Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pink enhances MLLM fine-grained perception by building diverse referential-comprehension instruction data from existing annotations (Visual Genome, Object365 via self-consistent bootstrapping) and tuning the visual encoder with Adapters/LoRA.
Problem
MLLMs have limited fine-grained image understanding; existing referential comprehension (RC) instruction data is narrow and GPT-4-generated data is costly and uncontrollable.
Approach
Convert existing dense annotations (Visual Genome relations/objects, Object365 boxes) into diverse RC instruction tasks, bootstrap referring expressions for boxes with self-consistent IoU filtering, and tune the visual encoder via lightweight Adapters instead of freezing or full-tuning it.
Key results
- Pink reaches 54.1% zero-shot RefCOCO val, versus 52.3% for Kosmos-2; with Object365 pairs (Pink*) it reaches 77.0%.
- With 6.7M tunable parameters and 396K instruction samples, Pink scores 47.8 on IconQA and 59.5 on OK-VQA, above Shikra-7B's 24.3 and 53.5.
- Pink achieves an overall MMBench CircularEval score of 74.1, the best among compared models (mPlug-Owl 68.5, Qwen-VL 61.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add visual relation reasoning, coarse spatial reasoning, object counting, and object detection tasks to the instruction tuning data (ablation) | Pink · PointQA-Local | VQA Score: 0.0% to 54.6% | Helped |
| Add visual relation reasoning, coarse spatial reasoning, object counting, and object detection tasks to the instruction tuning data (ablation) | Pink · VSR | VQA Score: 62.8 -> 66.3 | Helped |
| Add Object365 referring-expression-bounding-box pairs to instruction tuning (ablation) | Pink · RefCOCO val, zero-shot | Accuracy: 54.1% to 77.0% | Helped |
| Filter generated Object365 referring-expression-bounding-box pairs with the self-consistent method (ablation) | Pink with Object365 data · RefCOCO val, zero-shot | Accuracy: 73.8% to 77.0% | Helped |
| Freeze the visual encoder during instruction tuning (ablation) | Pink · RefCOCO val, zero-shot | Accuracy: 54.1% to 37.2% | Hurt |
| Fine-tune the entire visual encoder instead of using parameter-efficient tuning (ablation) | Pink · RefCOCO val, zero-shot | Accuracy: 54.1% to 0.05% | Hurt |
| Replace Adapter with LoRA for parameter-efficient tuning (ablation) | Pink · RefCOCO val, zero-shot | Accuracy: 54.1 -> 54.7 | No clear effect |
| Use Pink instead of Shikra-7B | Vicuna-7B-based multimodal model · VSR | VQA Score: 63.3 -> 66.3 | Helped |
| Use Pink instead of Qwen-VL | Multimodal model · MMBench test | Overall accuracy: 68.5 -> 74.1 | Helped |
Surprising findings
- Full-tuning the visual encoder collapses RefCOCO val accuracy from 54.1% to 0.05%.
- LoRA on the visual encoder slightly outperforms the Adapter variant on RefCOCO val (54.7 vs 54.1) despite the Adapter winning elsewhere.
Limitations
- Inherits biases and hallucination from the underlying LLM.
- Weak at detecting multiple tiny objects in complex scenes, possibly due to low input resolution (224x224).
Experimental setup
Datasets: LLaVA-CC3M-Pretrain-595K LLaVA-158K VQAv2 A-OKVQA Visual Genome Flickr30K Object365 RefCOCO RefCOCO+ RefCOCOg PointQA GQA
Benchmarks: VQAv2 IconQA VSR OK-VQA GQA RefCOCO RefCOCO+ RefCOCOg Visual-7W PointQA-Local PointQA-LookTwice MMBench
Metrics: accuracy VQA score
Baselines: Shikra-7B Qwen-VL Kosmos-2 Instruct-BLIP LLaVA-1.5 mPlug-Owl OFA-L
Builds on: LLaVA Shikra Adapter LoRA Vicuna
Closest papers by meaning
- Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception (CVPR 2024 · similarity 0.93)
- MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs (ICML 2026 · similarity 0.92)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.91)
- Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.91)
- SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models (CVPR 2024 · similarity 0.90)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.