Research map / Vision-language and multimodal / Multimodal LLMs
Towards Self-Refinement of Vision-Language Models with Triangular Consistency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SRF, a self-refinement framework in which a VLM (LLaVA-1.5) generates image-question-answer triplets from 1M unlabeled images, filters them with a Triangular Consistency score, and retrains without external supervision.
Problem
VLMs are usually improved with supervised visual instruction data that is costly, requires human annotation or stronger teacher VLMs; the question is whether a VLM can refine itself using only unlabeled images.
Approach
Multi-task instruction tuning (I→QA, IQ→A, IA→Q) lets the VLM generate QA pairs for unlabeled images; a triangular consistency score re-predicts masked Q or A and keeps the top 20% of generated triplets for retraining the base model on the merged data.
Key results
- SRF-LLaVA-1.5 7B reaches 66.9 on LLaVA-Bench and 33.1 on MM-Vet versus 63.4 and 30.5 for LLaVA-1.5 7B, with GQA 63.35 vs 62.0.
- SRF-LLaVA-1.5 13B exceeds LLaVA-1.5 13B on all 8 reported benchmarks (e.g., MME-C 334.2 vs 295.4).
- Retained triangular-consistency-filtered QA pairs score 85.3 (GPT) and 89.0 (human) versus 59.9 and 83.0 for excluded pairs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply self-refinement with triangular-consistency filtering | LLaVA-1.5 7B · LLaVA-Bench (In-the-Wild) | score: 63.4 -> 66.9 | Helped |
| Apply self-refinement with triangular-consistency filtering | LLaVA-1.5 7B · MM-Vet | score: 30.5 -> 33.1 | Helped |
| Apply self-refinement with triangular-consistency filtering | LLaVA-1.5 7B · MME Cognition | score: 316.1 -> 331.8 | Helped |
| Apply self-refinement with triangular-consistency filtering | LLaVA-1.5 7B · MMBench | score: 64.3 -> 66.3 | Helped |
| Apply self-refinement with triangular-consistency filtering | LLaVA-1.5 7B · ScienceQA-IMG | accuracy: 66.8 -> 67.72; Recap-LLaVA-1.5 7B: 69.01 | Mixed |
| Apply self-refinement | MobileVLM · MMBench | score: 53.2 -> 56.1 | Helped |
| Apply self-refinement | Qwen2.5-VL · MMBench | score: 81.5 -> 80.8 | Hurt |
| Use a second self-refinement round (ablation) | SRF-LLaVA-1.5 · MM-Vet | score: 33.1 -> 33.9 | Helped |
| Select the bottom 20% by consistency instead of the top 20% (ablation) | SRF-LLaVA-1.5 · MM-Vet | score: 33.1 -> 30.4 | Hurt |
| Set filtering threshold to top 5% instead of top 20% (ablation) | SRF-LLaVA-1.5 · MM-Vet | score: 33.1 -> 32.4 | Hurt |
| Set filtering threshold to top 50% instead of top 20% (ablation) | SRF-LLaVA-1.5 · MM-Vet | score: 33.1 -> 30.4 | Hurt |
| Set filtering threshold to top 80% instead of top 20% (ablation) | SRF-LLaVA-1.5 · MM-Vet | score: 33.1 -> 29.1 | Hurt |
Surprising findings
- Recap-LLaVA-1.5, which uses the same unlabeled images with unfiltered self-captions, yields only small gains, suggesting the filtering rather than extra data drives the improvement.
- Even without filtering (100% threshold) the model still improves by about 0.73%, indicating learning from new images even with noisy synthetic annotations.
Limitations
- Improvement diminishes when new images closely match the model's existing training distribution.
- Gains are modest; Round 2 yields diminishing returns.
- Self-generated data risks reinforcing biases and hallucinations; the causal assumptions cannot be directly verified for modern VLMs.
Experimental setup
Datasets: llava_v1_5_mix665k LAION
Benchmarks: LLaVA-Bench (In-the-Wild) MM-Vet MME MMBench MMBench-Chinese VQAv2 GQA ScienceQA-IMG MMMU MMMU-Pro MathVista MathVision
Metrics: accuracy score TTR Distinct-2
Baselines: LLaVA-1.5 7B LLaVA-1.5 13B Recap-LLaVA-1.5 InstructBLIP Qwen-VL-Chat MobileVLM Qwen2.5-VL
Builds on: LLaVA-1.5 visual instruction tuning Self-Refine BERTScore
Closest papers by meaning
- Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.92)
- SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models (CVPR 2024 · similarity 0.90)
- AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition (CVPR 2026 · similarity 0.90)
- Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.90)
- Unveiling the Tapestry of Consistency in Large Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.