Research map / Vision-language and multimodal / Multimodal LLMs
Enhancing Large Vision Language Models with Self-Training on Image Comprehension
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STIC, a two-stage self-training method that builds a self-generated image-description preference dataset from unlabeled MSCOCO images and then fine-tunes LVLMs on a small reused SFT subset infused with self-generated descriptions, improving LLaVA-v1.6 by 4.0% on average across seven benchmarks.
Problem
LVLMs need high-quality vision-language instruction data, which is costly to obtain; existing self-training methods from text-only LLMs do not address the image perception step that LVLM reasoning depends on.
Approach
Self-construct a DPO preference dataset for image description without image labels: preferred descriptions come from a step-by-step prompt, dispreferred ones from misleading prompts or corrupted images; add a positive-sample NLL regularizer; then recycle a small part of SFT data with self-generated descriptions prepended to instructions.
Key results
- STIC improves LLaVA-v1.6 (7B) average over seven benchmarks from 54.7% to 58.7% (+4.0%), with ScienceQA rising 68.9 to 75.3.
- On LLaVA-v1.5 (7B), STIC raises average from 45.3% to 47.0% and outperforms POVID on LLaVA-Bench (68.9 vs 68.7).
- Scaling preference data from 6k to 12k images raises LLaVA-Bench gain from 1.9% to 3.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply STIC self-training | LLaVA-v1.6 (7B) · ScienceQA | score: +6.4% | Helped |
| Apply STIC self-training | LLaVA-v1.6 (7B) · TextVQA | score: +4.9% | Helped |
| Apply STIC self-training | LLaVA-v1.6 (7B) · ChartQA | score: +5.1% | Helped |
| Apply STIC self-training | LLaVA-v1.6 (7B) · MathVista | score: +2.4% | Helped |
| Add describe-and-respond prompting (ablation) | Original LLaVA-v1.6 · seven vision-language benchmarks | average score: 2.3% | Mixed |
| Add describe-and-respond prompting (ablation) | STIC · seven vision-language benchmarks | average score: 1.1% | Helped |
| Remove dispreferred responses and use preferred responses for SFT (ablation) | STIC · LLaVA-Bench | score: 0.6% | Hurt |
| Increase Stage 1 preference images from 6k to 12k (ablation) | STIC · LLaVA-Bench | score gain: from 1.9% to 3.1% | Helped |
| Use VFLAN rather than COCO images for Stage 1 (ablation) | STIC · LLaVA-Bench | score: 79.2 -> 81.9 | Helped |
| Use a hallucination prompt instead of the normal prompt for DaR (ablation) | LLaVA-v1.6 (7B) · LLaVA-Bench | score: 78.5 -> 73.7 | Hurt |
Surprising findings
- Describe-and-respond prompting alone lowers the base LLaVA-v1.6 average by 2.3%, while combined with STIC fine-tuning it adds 1.1%.
- Removing dispreferred samples and using only SFT on preferred descriptions drops performance versus full STIC preference training.
Limitations
- Small gain on MathVista (2.4%) because preference data comes only from natural MSCOCO images.
- Preference data construction is fairly simple and may be insufficient for nuanced image detail understanding.
- Two-stage pipeline rather than end-to-end training; scaling limits not explored beyond 12k images.
Experimental setup
Datasets: MSCOCO train2014 LLaVA SFT data Vision-Flan (VFLAN)
Benchmarks: ScienceQA TextVQA ChartQA LLaVA-Bench MMBench MM-Vet MathVista
Metrics: accuracy average accuracy across benchmarks
Baselines: InstructBLIP mPLUG-OWL2 POVID LLaVA-v1.5 LLaVA-v1.6
Builds on: DPO LLaVA POVID self-training
Closest papers by meaning
- SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models (CVPR 2024 · similarity 0.92)
- Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · similarity 0.92)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.91)
- VladVA: Discriminative Fine-tuning of LVLMs (CVPR 2025 · similarity 0.91)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.90)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.