Research map / Vision-language and multimodal / Multimodal LLMs
Visual Instruction Bottleneck Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vittle, a variational information-bottleneck layer inserted inside an MLLM's LLM backbone during visual instruction tuning, improving robustness to input perturbations and long-tail queries with little overhead.
Problem
MLLMs degrade substantially on unfamiliar queries under distribution shifts (visual/textual perturbations and long-tail queries), and existing fixes require more instruction data or larger models.
Approach
Derives a variational lower bound of the information bottleneck objective for autoregressive MLLMs and implements it as a lightweight Gaussian bottleneck MLP inside the LLM (layer 24 of 32) that is trained end-to-end with a KL penalty, with an interpolated representation and fixed or learnable prior.
Key results
- On long-tail open-ended QA, Vittle (L) improves LB-Wild from 51.6 to 54.6 and LB-Wilder from 156.9 to 168.8 over the LLaVA baseline.
- On LB-COCO joint perturbations, Vittle (F) raises the score from 62.3 to 64.4 while keeping clean performance near baseline (76.1 vs 77.8).
- Vittle reduces representation JSD between clean and perturbed LB-COCO from 0.068 (baseline) to 0.047 (Vittle F) and EMID from 0.026 to 0.021 (Vittle L).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Vittle with a learnable Gaussian prior | LLaVA-v1.5-7B · LB-Wild, LB-Wilder, WV-Bench | relative preference score: 51.6 -> 54.6; 156.9 -> 168.8; 60.0 -> 60.4 | Helped |
| Add Vittle with a fixed standard Gaussian prior | LLaVA-v1.5-7B · LB-COCO V Pert., T Pert., and J Pert. | relative preference score: 73.4 -> 74.2; 72.2 -> 74.1; 62.3 -> 64.4 | Helped |
| Add Vittle with a learnable Gaussian prior | LLaVA-v1.5-7B · MMMU | accuracy: 35.6 -> 35.3 | Hurt |
| Add Vittle with a fixed standard Gaussian prior | LLaVA-v1.5-7B · MMMU | accuracy: 35.6 -> 34.5 | Hurt |
| Add Vittle with a learnable Gaussian prior | LLaVA-v1.5-7B · closed-form QA average | accuracy: 50.9 -> 51.1 | Helped |
| Add Vittle with a fixed standard Gaussian prior | LLaVA-v1.5-7B · closed-form QA average | accuracy: 50.9 -> 50.9 | No clear effect |
| Replace full fine-tuning with LoRA fine-tuning | LLaVA-v1.5-7B · LB-COCO Clean | relative preference score: 77.8 -> 73.4 | Hurt |
| Add Vittle bottleneck with a fixed standard Gaussian prior | LLaVA-v1.5-7B · LB-COCO variants | JSD: 0.068 -> 0.047 | Helped |
| Add Vittle bottleneck with a learnable Gaussian prior | LLaVA-v1.5-7B · LB-COCO variants | EMID: 0.026 -> 0.021 | Helped |
| Set representation interpolation coefficient alpha to 0.5 (ablation) | Vittle on LLaVA-v1.5-7B · POPE V Shifts Avg. | accuracy: 84.12 -> 84.90 | Helped |
| Set representation interpolation coefficient alpha to 0.25 (ablation) | Vittle on LLaVA-v1.5-7B · LB-COCO | relative preference score: 77.8 -> 75.6 | Hurt |
| Increase representation interpolation coefficient alpha to 0.75 or 1 (ablation) | Vittle on LLaVA-v1.5-7B · training | language modeling loss convergence | Hurt |
Surprising findings
- Information-maximization methods (ROSS, LIT) improve hallucination detection but underperform the baseline on open-ended QA under shifts.
- Applying the bottleneck to the penultimate layer or early layers gives worse results; early-layer bottleneck prevented language modeling loss convergence.
Limitations
- Relies on quality of gold responses, which are generated by another LLM and may be noisy.
- Slightly hurts OCR capability of the MLLM.
- IB alone does not guarantee generalization to counterfactual or domain-shift samples.
Experimental setup
Datasets: LLaVA-pretrain-558k LLaVA-mix-665k LLaVA-Bench-COCO POPE ScienceQA MMMU MME MMStar LLaVA-Bench in-the-wild LLaVA-Bench-Wilder WildVision-Bench
Benchmarks: LLaVA-Bench-COCO perturbed variants POPE perturbed variants LLaVA-Bench in-the-wild LLaVA-Bench-Wilder WildVision-Bench ScienceQA MMMU MME MMStar
Metrics: relative preference score exact match accuracy JSD EMID
Baselines: LLaVA-v1.5 LoRA weight decay ROSS LIT
Builds on: LLaVA-v1.5 information bottleneck variational information bottleneck LoRA ROSS
Closest papers by meaning
- Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning (ICML 2025 · similarity 0.89)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.89)
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.88)
- Efficient Large Multi-modal Models via Visual Context Compression (NeurIPS 2024 · similarity 0.88)
- Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.