Research map / Vision-language and multimodal / Multimodal LLMs
SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SC-Tune, a PPO-based describer-locator cyclic fine-tuning of object-level LVLMs (Qwen-VL, MiniGPT-v2) using bbox-caption-bbox IoU self-consistency rewards, improving referential comprehension and self-consistency on out-of-domain data.
Problem
Object-level LVLMs show poor self-consistency in bbox-caption-bbox cycles (generate a referring expression for a box, then re-localize it), especially on out-of-domain images, indicating weak generalization of referential comprehension.
Approach
Treat one pretrained LVLM as two roles, a describer (captioner trained with PPO using IoU of the locator's re-predicted box as reward plus a KL penalty) and a locator (trained with MLE on describer-generated captions), alternating freezing and parameter synchronization like a DQN target network, so captions become discriminative and localization generalizes.
Key results
- On Object365 self-consistency, Qwen-VL rises from 76.9 to 94.1 Pr@0.5 with SC-Tune, and MiniGPT-v2 from 65.4 to 76.6.
- On ReferItGame REC, Qwen-VL improves from 61.48 to 69.28 and MiniGPT-v2 from 36.05 to 43.68 accuracy.
- On ReferItGame REG, Qwen-VL CIDEr test A rises from 19.2 to 43.2 under the out-of-domain setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply SC-Tune trained on Object365 | Qwen-VL · Object365 self-consistency | accuracy: 76.9 -> 94.1 | Helped |
| apply SC-Tune trained on Object365 | Qwen-VL · OpenImages self-consistency | accuracy: 52.9 -> 68.8 | Helped |
| apply SC-Tune | Qwen-VL · ReferItGame REC | accuracy: 61.48 -> 69.28 | Helped |
| apply SC-Tune | Qwen-VL · Flickr30K REC | accuracy: 64.15 -> 77.49 | Helped |
| apply SC-Tune | Qwen-VL · ReferItGame test A REG | CIDEr: 19.2 -> 43.2 | Helped |
| apply SC-Tune | Qwen-VL · Visual-7W | accuracy: 34.77 -> 46.09 | Helped |
| apply SC-Tune | Qwen-VL · RefCOCO val REC | accuracy: 88.88 -> 88.04 | Hurt |
| add supervised RefCOCO data during locator training (ablation) | Qwen-VL + SC-Tune · RefCOCO val REC | accuracy: 88.04 -> 89.61 | Helped |
| tune both locator and describer instead of locator only (ablation) | Qwen-VL · ReferItGame REC | accuracy: 64.82 -> 69.28 | Helped |
| tune both locator and describer instead of describer only (ablation) | Qwen-VL · ReferItGame test A REG | CIDEr: 38.2 -> 43.2 | Helped |
| use 200 steps per cycle instead of 50 (ablation) | Qwen-VL + SC-Tune · ReferItGame test A REG | CIDEr: 34.3 -> 43.2 | Helped |
| use 1000 steps per cycle instead of 200 (ablation) | Qwen-VL + SC-Tune · ReferItGame REC | accuracy: 69.28 -> 67.45 | Hurt |
Surprising findings
- SC-Tune slightly reduces in-domain RefCOCO REC accuracy for both base models (e.g., Qwen-VL val 88.88 to 88.04) before the RefCOCO supervision is added back.
- Self-consistency on out-of-domain OpenImages was much lower than on in-domain RefCOCO for existing LVLMs (e.g., Qwen-VL 52.9 vs 87.9).
Limitations
- In-domain RefCOCO REC performance drops slightly after SC-Tune unless in-domain supervised data is mixed into locator training.
- Evaluated only on two LVLMs (Qwen-VL and MiniGPT-v2) with a single fine-tuning run per setting.
Experimental setup
Datasets: Object365 OpenImages Visual Genome RefCOCO
Benchmarks: RefCOCO RefCOCO+ RefCOCOg ReferItGame Flickr30K Entities Visual7W PointQA-Local NoCaps Flickr30K VQAv2 GQA
Metrics: accuracy (Pr@0.5) METEOR CIDEr
Baselines: Qwen-VL MiniGPT-v2 SeqTR VLTVG CLIP-VG PFOS Shikra-7B BLIP-2 InstructBLIP
Builds on: PPO REINFORCE with baseline DQN target network RLHF
Closest papers by meaning
- Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.92)
- Unveiling the Tapestry of Consistency in Large Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · similarity 0.90)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.90)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.