Research map / Vision-language and multimodal / Visual question answering
BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BLIVA, which augments InstructBLIP's Q-Former query embeddings with directly projected encoded image patch embeddings fed to the LLM, improving text-rich and general VQA, and releases a YouTube thumbnail VQA dataset (YTTB-VQA).
Problem
Vision-language models with a fixed small set of learned query embeddings lose fine-grained information, which hurts reading and reasoning over text-rich images.
Approach
Combine learned query embeddings from a Q-Former with a linear projection of raw encoded patch embeddings from the frozen vision encoder, concatenating both as soft prompts to the LLM, with a two-stage training scheme (patch-projection pretraining on 558K caption pairs, then instruction tuning of Q-Former and projection with frozen encoder and LLM).
Key results
- BLIVA (Vicuna-7B) averages 21.75 on text-rich OCR benchmarks vs 19.22 for InstructBLIP (Vicuna-7B), with OCR-VQA 65.38 vs 47.62.
- BLIVA achieves an MME total of 1669.2 vs 1417.9 for InstructBLIP, and average 119.2 vs 101.3.
- On YTTB-VQA thumbnails, BLIVA reaches 83.5% accuracy vs 82.2% for InstructBLIP (Vicuna-7B).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add projected image patch embeddings alongside query embeddings | InstructBLIP (Vicuna-7B) · Text-rich VQA average | accuracy (%): 19.22 -> 21.75 | Helped |
| Add projected image patch embeddings (ablation) | InstructBLIP Baseline (Vicuna-7B) · TextVQA | accuracy (%): 37.58 -> 58.8 | Helped |
| Add projected image patch embeddings (ablation) | InstructBLIP Baseline (Vicuna-7B) · Hateful Memes | AUC: 50.6 -> 49.1 | Hurt |
| Add projected image patch embeddings (ablation) | InstructBLIP Baseline (Vicuna-7B) · VizWiz | accuracy (%): 44.1 -> 42.83 | Hurt |
| Add patch-embedding pre-training (ablation) | BLIVA with patch embeddings · VSR | accuracy (%): 58.85 -> 62.2 | Helped |
| Add patch-embedding pre-training (ablation) | BLIVA with patch embeddings · TextVQA | accuracy (%): 58.8 -> 57.96 | Hurt |
| Fine-tune the LLM with LoRA during instruction tuning (ablation) | BLIVA with patch pre-training · VSR | accuracy (%): 62.2 -> 51.39 | Hurt |
| Replace linear projection with convolutional position encoding (ablation) | BLIVA visual assistant branch · Flickr30K | CIDEr: 87.1 -> 85.87 | Hurt |
| Replace linear projection with ViT block and convolutional position encoding (ablation) | BLIVA visual assistant branch · VSR | accuracy (%): 62.2 -> 63.90 | Helped |
| Replace linear projection with ViT block and convolutional position encoding (ablation) | BLIVA visual assistant branch · TextVQA | accuracy (%): 42.18 -> 40.72 | Hurt |
| Replace linear projection with ViT block and relative position encoding (ablation) | BLIVA visual assistant branch · TextVQA | accuracy (%): 42.18 -> 41.24 | Hurt |
| Replace linear projection with an MLP layer (ablation) | BLIVA visual assistant branch · Flickr30K | CIDEr: 87.1 -> 82.9 | Hurt |
Surprising findings
- Without the pre-training stage, BLIVA's gains on ST-VQA, OCR-VQA, TextVQA, ESTVQA and POIE decreased, and fine-tuning the LLM with LoRA reduced performance on many tasks.
- Adding patch embeddings did not improve HM or VizWiz, and BLIVA (FlanT5-XXL) scored lower than InstructBLIP on Visual Dialog (36.18 vs 48.5).
Limitations
- Struggles with deciphering numerical symbols in images (human evaluation).
- Lower performance on information-heavy VQA (InfoVQA, ChartQA, ESTVQA) attributed to small pretraining set.
- Loss of internet/world knowledge (artwork, landmark) attributed to lightweight pretraining and missing LAION-115M data.
Experimental setup
Datasets: LAION-558K CC-3M SBU MSCOCO TextCaps VQAv2 OKVQA A-OKVQA OCR-VQA LLaVA-Instruct-150K TextVQA ST-VQA
Benchmarks: STVQA OCRVQA TextVQA DocVQA InfoVQA ChartQA ESTVQA FUNSD SROIE POIE VSR IconQA
Metrics: accuracy CIDEr AUC MRR MME perception and cognition score
Baselines: InstructBLIP BLIP-2 LLaVA MiniGPT-4 mPLUG-Owl OpenFlamingo Flamingo
Builds on: InstructBLIP BLIP-2 LLaVA
Closest papers by meaning
- Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective (AAAI 2025 · similarity 0.90)
- FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering (NeurIPS 2025 · similarity 0.90)
- VTQA: Visual Text Question Answering via Entity Alignment and Cross-Media Reasoning (CVPR 2024 · similarity 0.90)
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.90)
- Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark (ICLR 2025 · similarity 0.90)
- Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.