Research map / Vision-language and multimodal / Visual question answering
Self-Training Large Language Models for Improved Visual Program Synthesis With Visual Reinforcement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VisReP, an offline reinforced self-training recipe that uses binary coarse rewards from existing vision-language annotations to fine-tune CodeLlama-7B (via LoRA) to write better ViperGPT-style visual programs.
Problem
Visual program synthesis with LLMs relies on few-shot prompting of frozen LLMs, and there is no large dataset of ground-truth visual programs for supervised training of an open LLM.
Approach
Use the answer-correctness of executed programs against existing task annotations as a binary coarse reward, and improve the LLM policy by reward-filtered behavioral cloning (grow/improve iterations of REINFORCE-style self-training), with human-written corrections as in-context examples to stabilize iterated training.
Key results
- On GQA, self-trained CodeLlama-7B reaches 59.2% vs 50.0% for frozen CodeLlama-7B and 53.9% for gpt-3.5-turbo.
- On SugarCREPE, VisReP improves CodeLlama-7B from 43.5% to 58.7%, above gpt-3.5-turbo's 48.9%.
- Without human corrections, iterated self-training degrades over iterations; with a small set of human-written in-context corrections it stabilizes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| self-train CodeLlama-7B with VisReP | CodeLlama-7B · GQA | accuracy: 50.0 ± 1.7 -> 59.2 ± 1.4 | Helped |
| self-train CodeLlama-7B with VisReP | CodeLlama-7B · Omnilabel | Macro-F1: 37.3 ± 1.6 -> 42.4 ± 1.0 | Helped |
| self-train CodeLlama-7B with VisReP | CodeLlama-7B · Omnilabel-Hard | Macro-F1: 23.7 ± 1.4 -> 28.1 ± 0.9 | Helped |
| self-train CodeLlama-7B with VisReP | CodeLlama-7B · Winoground | accuracy: 41.3 ± 0.03 -> 52.7 ± 0.6 | Helped |
| self-train CodeLlama-7B with VisReP | CodeLlama-7B · SugarCrepe | accuracy: 43.5 ± 0.8 -> 58.7 ± 1.5 | Helped |
| self-train on GQA with VisReP | CodeLlama-7B · VQAv2 | accuracy: 46.6 ± 1.1 -> 61.0 ± 1.0 | Helped |
| self-train on GQA with VisReP | CodeLlama-7B · OK-VQA | accuracy: 12.6 ± 2.1 -> 33.7 ± 1.9 | Helped |
| self-train on GQA with VisReP | CodeLlama-7B · Omnilabel | Macro-F1: 37.3 ± 1.6 -> 39.9 ± 0.7 | Helped |
| self-train on GQA with VisReP | CodeLlama-7B · SugarCrepe | accuracy: 43.5 ± 0.8 -> 51.0 ± 1.5 | Helped |
| add human-written corrections as in-context examples for hard question types (ablation) | VisReP on GQA · GQA | validation accuracy | Helped |
| iterate self-training without human-written corrections (ablation) | VisReP on GQA · GQA | validation accuracy | Hurt |
| reduce training examples from 100 to 10 per question type (ablation) | VisReP on GQA · GQA | validation accuracy | Helped |
Surprising findings
- Iterated self-training without human corrections degrades performance because programs that are right for the wrong reasons get reinforced.
- Self-training still improves performance when training data is reduced to about 1k samples, and multiple tries per query can mitigate data scarcity.
Limitations
- Coarse binary reward can reinforce programs that are correct for the wrong reasons.
- Iterated self-training degrades without human-written corrections for persistent errors.
- Performance is bounded by the available perception modules (e.g., BLIP, GroundingDINO), which struggle on compositional image-text matching.
Experimental setup
Datasets: GQA OmniLabel SugarCrepe WinoGround VQAv2 OK-VQA
Benchmarks: GQA OmniLabel OmniLabel-Hard WinoGround SugarCREPE VQAv2 OK-VQA
Metrics: accuracy macro-F1
Baselines: ViperGPT gpt-3.5-turbo frozen CodeLlama-7B VisProg CodeVQA
Builds on: ReST ViperGPT REINFORCE CodeLlama RaFT
Closest papers by meaning
- Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.89)
- Towards Self-Refinement of Vision-Language Models with Triangular Consistency (NeurIPS 2025 · similarity 0.89)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.88)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.88)
- CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding (ICLR 2024 · similarity 0.88)
- VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.