Research map / Vision-language and multimodal / CLIP and zero-shot
Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VPG-C, a lightweight module that uses intermediate LLM states to guide a Q-Former to recover missing visual details, trained with a synthetic discriminative (image-edit difference description) strategy, and introduces the DEMON benchmark for demonstrative instruction following.
Problem
Visual Prompt Generators in MLLMs are trained with image captioning, which biases them toward primary content and omits details needed to understand demonstrative (interleaved multi-image, multimodal) instructions.
Approach
Insert a conditional re-extraction step at an intermediate LLM layer: the LLM's hidden state of the last token forms instruction-specific guidance that queries the Q-Former to recover residual visual details, which are added back via a zero-initialized skip connection; the guidance module is tuned on synthetic image-edit pairs where the model must describe the difference, avoiding supervised demon
Key results
- VPG-C achieves the highest average on every DEMON task category versus the listed baselines; e.g., multimodal dialogue 37.50 vs 33.58 for InstructBLIP.
- On MME, VPG-C scores 1299.24 perception and 321.07 cognition, above the best baselines (1293.84 BLIP-2 and 306.43 Otter).
- Ablation shows synthetic training raises average DEMON scores over VPG-C without synthetic training (e.g., multimodal dialogue 31.76 to 37.50, knowledge grounded QA 42.53 to 48.60).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add heuristic-extracted, less-attended visual details (ablation) | Backbone VPG · DEMON Multimodal Dialogue | average score: 25.65 -> 28.13 | Helped |
| replace heuristic details with VPG-C (ablation) | Backbone VPG · DEMON Visual Relation Inference | average score: 12.69 -> 19.12 | Helped |
| add synthetic discriminative training to VPG-C (ablation) | VPG-C · DEMON Visual Relation Inference | average score: 19.12 -> 25.90 | Helped |
| condition InstructBLIP feature extraction on its initially generated answer (ablation) | InstructBLIP · DEMON Multimodal Dialogue | average score: 33.58 -> 32.10 | Hurt |
| add synthetic training to the answer-condition InstructBLIP variant (ablation) | InstructBLIP + Answer Condition · DEMON Multimodal Dialogue | average score: 32.10 -> 31.76 | Hurt |
| add VPG-C with a linear visual prompt generator (ablation) | LLaVA · DEMON Multimodal Dialogue | average score: 7.79 -> 16.43 | Helped |
| increase synthetic training data from 64K to 80K examples (ablation) | VPG-C · Synthetic training data efficiency | Accuracy: 41.49 -> 41.62 | Helped |
| increase synthetic training data from 80K to 96K examples (ablation) | VPG-C · Synthetic training data efficiency | Accuracy: 41.62 -> 40.12 | Hurt |
Surprising findings
- Naively adding InstructBLIP's answer as a condition for re-extracting visual features, or fine-tuning with the same synthetic data, did not give notable gains over InstructBLIP.
- Most SOTA models were found to be insensitive to the order of interleaved images in demonstrative instructions.
Limitations
- Image-choice cloze tasks remain largely unsolved by all evaluated models.
- Synthetic training data beyond 64K led to overfitting, so gains depend on data scale tuning.
Experimental setup
Datasets: CC3M DEMON benchmark source datasets (e.g., ALFRED, VIST, MME-related sources)
Benchmarks: DEMON MME OwlEval
Metrics: ROUGE-L accuracy human rating (1-5)
Baselines: BLIP-2 InstructBLIP LLaMA-Adapter V2 LLaVA MiniGPT-4 mPLUG-Owl OpenFlamingo Otter
Builds on: BLIP-2 InstructBLIP LLaVA Blended Diffusion
Closest papers by meaning
- VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.89)
- VILA: On Pre-training for Visual Language Models (CVPR 2024 · similarity 0.88)
- ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts (CVPR 2024 · similarity 0.88)
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.87)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- Learning by Correction: Efficient Tuning Task for Zero-Shot Generative Vision-Language Reasoning (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.