Research map / Vision-language and multimodal / Cross-modal learning
Towards Better Vision-Inspired Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VIVL, a vision-language connection module with a feature pyramid extractor over intermediate ViT layers and deep vision-conditioned prompts inside the LLM, improving captioning, VQA and ScienceQA when trained from scratch or plugged into BLIP-2 and LLaVA.
Problem
Existing VL connection modules use only the last layer of the vision encoder and only shallow vision-language interaction in the LLM, leaving visual detail underexploited and requiring large pre-training for good data efficiency.
Approach
A feature pyramid extractor fuses intermediate-layer ViT features via cross-attention with learnable queries, and deep vision-conditioned prompts (DVCP) regenerate prompts at each LLM layer from vision features and previous-layer outputs using a shared cross-attention module applied every S layers.
Key results
- Trained from scratch on COCO caption, VIVL reaches 140.3 CIDEr versus 122.2 for LLaMA-Adapter V2 (+18.1).
- BLIP-2+VIVL reaches 85.00 VQAv2 test-dev accuracy with OPT2.7B, +3.41 over BLIP-2 baseline (81.59).
- LLaVA+VIVL reaches 91.51% on ScienceQA average, +0.59 over LLaVA; image-context accuracy gains 1.24 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VIVL instead of LLaMA-Adapter V2 when training the connection module from scratch | COCO Caption · COCO Caption | CIDEr: 122.2 -> 140.3; 18.1 CIDEr gains | Helped |
| Add VIVL to LLaVA | LLaVA · Science QA image context | accuracy: 88.00 -> 89.24 | Helped |
| Add VIVL to BLIP-2 | BLIP-2 ViT-g + FlanT5XL · COCO Caption Karpathy test | CIDEr: 144.5 -> 145.6 (+1.1) | Helped |
| Add VIVL to BLIP-2 | BLIP-2 ViT-g + FlanT5XL · NoCaps zero-shot | CIDEr: 121.6 -> 122.7 (+1.1) | Helped |
| Add VIVL to BLIP-2 | BLIP-2 ViT-g + OPT2.7B · VQAv2 val | accuracy: 81.59 -> 85.00 (+3.41) | Helped |
| Add FPE (ablation) | BLIP-2 OPT2.7B · NoCaps zero-shot | CIDEr: 120.0 -> 120.6 | Helped |
| Add DVCP (ablation) | BLIP-2 OPT2.7B · NoCaps zero-shot | CIDEr: 120.0 -> 121.1 | Helped |
| Use FPE and DVCP together (ablation) | BLIP-2 OPT2.7B · NoCaps zero-shot | CIDEr: 120.0 -> 121.3 | Helped |
| Use FPE with multiple selected visual layers (ablation) | BLIP-2 ViT-g + OPT2.7B · COCO Caption | CIDEr: 141.2 -> 143.1 | Helped |
| Replace FPE with feature ensemble using layers [25, 38] (ablation) | BLIP-2 ViT-g + OPT2.7B · COCO Caption | CIDEr: 141.2 -> 141.9 | Helped |
| Use P-Tuning instead of DVCP (ablation) | BLIP-2 ViT-g + OPT2.7B · COCO Caption | CIDEr: 141.2 -> 139.8 | Hurt |
| Use P-Tuning v2 instead of DVCP (ablation) | BLIP-2 ViT-g + OPT2.7B · COCO Caption | CIDEr: 141.2 -> 139.0 | Hurt |
Surprising findings
- Simply increasing the number of prompt parameters (P-Tuning, P-Tuning v2) slightly hurts BLIP-2 on COCO caption, indicating vision-conditioning of prompts matters more than parameter count.
Limitations
- Pre-training with large-scale data was not explored due to resource constraints.
- Frozen-LLM setting limits what the approach has been tested on; gains on fine-tuned LLMs are not reported.
Experimental setup
Datasets: COCO Caption VQAv2 ScienceQA NoCaps
Benchmarks: COCO Caption NoCaps VQAv2 ScienceQA
Metrics: BLEU@4 CIDEr SPICE accuracy
Baselines: BLIP-2 LLaVA LLaMA-Adapter V2 BLIP ClipCap OSCAR VinVL OFA Flamingo SimVLM BEiT-3 P-Tuning
Builds on: BLIP-2 LLaVA P-Tuning v2 CoCoOp Feature Pyramid Networks
Closest papers by meaning
- Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Probing Visual Language Priors in VLMs (ICML 2025 · similarity 0.89)
- Vision-Language Model Selection and Reuse for Downstream Adaptation (ICML 2025 · similarity 0.89)
- Language Models as Black-Box Optimizers for Vision-Language Models (CVPR 2024 · similarity 0.88)
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.88)
- MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.