Research map / Vision-language and multimodal / Multimodal LLMs
Learning to Instruct for Visual Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes L2T, which extends visual instruction tuning by also computing loss on the instruction tokens (with templates removed), acting as a regularizer that reduces shortcut learning and overfitting in MLLMs.
Problem
Visual instruction tuning (VIT) of multimodal LLMs leads to overfitting and shortcut learning, where models rely on language priors and under-use visual input, hurting performance, captioning, and hallucination robustness.
Approach
Add a negative log-likelihood loss over the user instruction tokens (conditioned on the image) alongside the response loss, after removing system and frequent task-template tokens, so the model must learn to generate instructions from images, which regularizes it toward visual content at negligible cost.
Key results
- L2T improves overall relative performance over VIT across 16 benchmarks by 5.6% to 8.5% depending on model, with captioning gains up to 17.6% and OCR gains up to 8.8%.
- On MMHAL-Bench, L2T raises the GPT score from 1.73 to 2.36 and lowers hallucination rate from 0.68 to 0.53 for LLaVA-1.5 Vicuna-7B.
- L2T adds negligible training overhead (0.331 vs 0.334 steps per second for VIT).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| learn to generate image-grounded instructions in addition to responses | VIT · OCR Bench | accuracy: +8.8% | Helped |
| learn to generate image-grounded instructions in addition to responses | VIT · image captioning benchmarks | CIDEr: +17.6% | Helped |
| learn to generate image-grounded instructions in addition to responses | VIT · MMHAL-Bench | GPT score: 1.73 -> 2.36 | Helped |
| learn to generate image-grounded instructions in addition to responses | VIT · MMHAL-Bench | hallucination rate: 0.68 -> 0.53 | Helped |
| learn instructions including system and task templates, as well as answers (ablation) | L2T · VQAv2-L | accuracy: 64.02 -> 65.06 | Helped |
| remove system templates from learned instructions (ablation) | L2T · VQAv2-L | accuracy: 65.06 -> 65.26 | Helped |
| remove system and task templates from learned instructions (ablation) | L2T · VQAv2-L | accuracy: 65.26 -> 66.34 | Helped |
| increase instruction data from 40% to 80% (ablation) | L2T · eight evaluation metrics | average relative performance: 5.9%, 13.0%, and 9.3% | Helped |
| use 80% rather than 100% of pretraining data (ablation) | L2T · pretraining data scale evaluation | performance: 10.9%, 8.7%, and 8.5% | Helped |
| use QA rather than grounding instruction data (ablation) | L2T · VQAv2-L | accuracy: 53.10 -> 52.62; 63.26 -> 62.28 | Mixed |
| add 100k model-generated instruction–response pairs (ablation) | L2T TinyLLaVA-0.5B · VQAv2-L | accuracy: 56.68 -> 60.80 | Helped |
Surprising findings
- Training on the instruction loss makes the model slightly worse on the training loss (VQAv2) but better on unseen test data (DocVQA, NoCaps), indicating reduced overfitting.
- Grounding-only instruction data yields almost no gain from L2T, since its instructions are fixed templates with random bounding box coordinates.
Limitations
- Gains depend on instruction informativeness; templated or redundant instructions contribute nothing.
- Instruction learning may propagate harmful or biased content present in training instructions.
- Evaluation of hallucination relies partly on GPT-4 judgments; some benchmark gains are small (e.g., HallusionBench qAcc +0.44%).
Experimental setup
Datasets: LLaVA-pretrain-558k LLaVA-mix-665k LLaVA-NeXT-Data VQAv2 GQA
Benchmarks: VQAv2 GQA ScienceQA VizWiz MME MMMU MMStar ChartQA TextVQA DocVQA OCR Bench COCO2017
Metrics: accuracy CIDEr MME perception score CHAIR_i CHAIR_s GPT score hallucination rate qAcc fAcc aAcc
Baselines: VIT (standard visual instruction tuning) Prism-7B IM
Builds on: LLaVA TinyLLaVA LLaVA-NeXT Instruction Modelling (IM)
Closest papers by meaning
- Reconstructive Visual Instruction Tuning (ICLR 2025 · similarity 0.91)
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.90)
- Improved Baselines with Visual Instruction Tuning (CVPR 2024 · similarity 0.90)
- Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection (ICLR 2025 · similarity 0.90)
- Instruction-Guided Visual Masking (NeurIPS 2024 · similarity 0.89)
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.