Research map / Vision-language and multimodal / Human and embodied modeling
Question Aware Vision Transformer for Multimodal Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QA-ViT, which injects question representations into the top layers of a frozen ViT via gated cross-attention-style self-attention fusion, making visual encoding question-aware across VL architectures.
Problem
Vision encoders in VL models encode images independently of the user question, so visual features may miss query-relevant fine-grained details, especially for text-in-image and small-object questions.
Approach
Concatenate projected question features into the self-attention of the top L frozen ViT layers so image tokens attend to the question, with a parallel zero-initialized gated projection (tanh(beta)) that preserves the original features at initialization; the method is model-agnostic and modifies only the vision encoder.
Key results
- ViT+T5-large with QA-ViT improves VQAv2 from 70.0 to 72.0 and TextVQA from 44.7 to 48.7 (+4.0).
- LLaVA-1.5 + QA-ViT reaches 80.5 on VQAv2 and 59.1 on TextVQA, and 36.5 on VizWiz (zero-shot) vs 33.9 baseline.
- InstructBLIP-xl + QA-ViT matches or beats InstructBLIP-xxl on general and scene-text averages (106.5 / 69.2 vs 106.1 / 68.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add QA-ViT | LLaVA-1.5 · VQAv2 | vqa-score: +0.8 | Helped |
| Add QA-ViT | LLaVA-1.5 · VQAT | vqa-score: +1.7 | Helped |
| Add QA-ViT | LLaVA-1.5 · TextCaps | CIDEr: +2.3 | Helped |
| Add QA-ViT | LLaVA-1.5 · VizWiz | vqa-score: +2.6 | Helped |
| Replace question conditioning with a fixed textual prompt (ablation) | ViT+T5-large · VQAv2 and VQAT | accuracy: +0.1%; +1.1% | Mixed |
| Finetune the full vision encoder (ablation) | ViT+T5-large · VQAv2 and VQAT | accuracy: -0.5%; +0.2% | Mixed |
| Move question fusion to early ViT layers (ablation) | QA-ViT · VQAv2 and VQAT | accuracy: -2.1%; -3.0% | Hurt |
| Apply question fusion every 2 layers (ablation) | QA-ViT · VQAv2 and VQAT | accuracy: +0.7%; +1.9% | Helped |
| Apply question fusion at every ViT layer (ablation) | QA-ViT · VQAv2 and VQAT | accuracy: -0.5%; +1.2% | Mixed |
| Use LLM embeddings instead of the LLM encoder for question representations (ablation) | QA-ViT · VQAv2 and VQAT | accuracy: +1.0%; +2.8% | Helped |
| Use BERT for question representations (ablation) | QA-ViT · VQAv2 and VQAT | accuracy: +1.8%; +3.6% | Helped |
| Use the CLIP text encoder for question representations (ablation) | QA-ViT · VQAv2 and VQAT | accuracy: +1.8%; +3.3% | Helped |
Surprising findings
- Full finetuning of the CLIP vision encoder improves TextVQA but reduces VQAv2, indicating training instability.
- Applying QA-ViT in early layers degrades performance, while late-layer fusion works best.
Limitations
- Gains are modest for larger models (e.g., +0.3 to +0.8 on some general benchmarks)
- Evaluated only with models trained on the authors' own mixed dataset; no comparison with strong instruction-tuned general models beyond LLaVA-1.5 under equal data
Experimental setup
Datasets: VQAv2 COCO Captions TextCaps TextVQA ST-VQA OCR-VQA DocVQA VizWiz
Benchmarks: VQAv2 COCO Captions TextVQA ST-VQA TextCaps VizWiz
Metrics: VQA score CIDEr ANLS
Baselines: ViT+T5 BLIP2 InstructBLIP LLaVA-1.5 mPLUG-DocOwl OpenFlamingo-9B IDEFICS-9B IDEFICS-80B Shikra Qwen-VL InstructBLIP+OCR
Builds on: CLIP InstructBLIP BLIP-2 LLaVA-1.5 Flamingo-style gated cross-attention
Closest papers by meaning
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.90)
- BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions (AAAI 2024 · similarity 0.90)
- Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models (CVPR 2025 · similarity 0.90)
- Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping (ICLR 2025 · similarity 0.89)
- Latent Visual Reasoning (ICLR 2026 · similarity 0.89)
- Long Grounded Thoughts: Synthesizing Visual Problems and Reasoning Chains at Scale (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.