Research map / Vision-language and multimodal / Visual question answering
Multimodal Hypothetical Summary for Retrieval-based Multi-image Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multimodal Hypothetical Summary (MHyS), question- and description-form text summaries generated from images by an MLLM and an LLM-based QAG model, so retrieval becomes text-to-text and is coupled with VQA for retrieval-based multi-image QA on RETVQA.
Problem
Retrieval-based multi-image question answering suffers from cascading errors in "retrieve-then-answer" pipelines because the QA training objective does not optimize retrieval, and cross-modal retrieval has a modality gap.
Approach
Replace real images in retrieval with MHyS generated from images (question-form from visual and textual perspectives, plus detailed descriptions), enabling text-to-text retrieval; score with coarse-to-fine sentence-level (CLIP) plus word-level (VL-BART-based encoder) similarity, and jointly train retrieval (contrastive loss) and QA (VQA loss) end-to-end.
Key results
- Achieves 80.3% accuracy on RETVQA multi-image QA, a 3.7% absolute improvement over MI-BART (76.5%).
- Improves over CLIP-VLBART (65.8%) by 14.5 points in the multi-image setting.
- In the top-1 single-image setting, reaches 74.72% vs 64.39% for CLIP-VLBART.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add description-form MHyS to question-form MHyS (ablation) | MHyS (Ours) · RETVQA | accuracy: +9.99% | Helped |
| Add contrastive enhancement loss to align the query with MHyS (ablation) | MHyS (Ours) · RETVQA | accuracy: +0.49% | Helped |
| Use question-form MHyS instead of description-form MHyS (ablation) | MHyS · RETVQA | accuracy: +7.58% | Helped |
| Generate question-form MHyS from the textual scene perspective rather than the visual perspective (ablation) | MHyS · RETVQA | accuracy: +1.85% | Helped |
| Combine scene-related and image-related question MHyS (ablation) | MHyS · RETVQA | accuracy: +8.25% | Helped |
| Use detailed descriptions rather than captions (ablation) | MHyS · RETVQA | accuracy | Helped |
| Use word-level MHyS similarity rather than sentence-level CLIP similarity (ablation) | retrieval similarity · RETVQA | accuracy: +16.4% | Helped |
| Combine sentence-level CLIP similarity with log-transformed word-level MHyS similarity (ablation) | retrieval similarity · RETVQA | accuracy: +16.61% ~ +0.21% | Helped |
| Use log-transformed similarity for both CLIP and MHyS (ablation) | retrieval similarity · RETVQA | accuracy | Hurt |
| Use CLIP plus log-transformed MHyS similarity instead of sentence-level CLIP alone (ablation) | retrieval similarity · RETVQA | accuracy | Helped |
| Use direct addition of CLIP and MHyS similarity (ablation) | retrieval similarity · RETVQA | accuracy | Helped |
| Use adaptive fusion of CLIP and MHyS similarities (ablation) | retrieval similarity · RETVQA | accuracy | Helped |
Surprising findings
- Two-stage retrieve-then-answer methods outperform introduce-retrieval-into-QA methods in multi-image QA, but the reverse holds for top-1 single-image QA.
- Fine-tuned or zero-shot MLLMs (mPLUG-Owl2, Qwen-VL) perform worse on multi-image QA than on single-image QA, despite being strong generalists.
Limitations
- Relies on a fixed number of retrieved images (best at five); too few or too many images hurt performance.
- Evaluated only on the RETVQA dataset; generalization to other benchmarks is not shown.
- Requires generating MHyS for every image with a multimodal LLM, adding preprocessing cost.
Experimental setup
Datasets: RETVQA
Benchmarks: RETVQA
Metrics: accuracy
Baselines: MI-BART MI-VLBART MI-VLP MI-(LSTM+VGG) mPLUG-VLBART CLIP-VLBART CLIP-BAN CLIP-LXMERT mPLUG-QWENVL mPLUG-Owl2 LLaVA-1.5
Builds on: CLIP VL-BART mPLUG-Owl2 MI-BART hypothetical document embedding
Closest papers by meaning
- Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark (ICLR 2025 · similarity 0.89)
- Benchmarking Retrieval-Augmented Multimomal Generation for Document Question Answering (NeurIPS 2025 · similarity 0.89)
- MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.89)
- DIEM: Decomposition-Integration Enhancing Multimodal Insights (CVPR 2024 · similarity 0.88)
- Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering (CVPR 2025 · similarity 0.88)
- Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.