Research map / Vision-language and multimodal / Visual question answering
How to Configure Good In-Context Sequence for Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how to configure in-context demonstrations (retrieval by image/question/answer, triplet mismatching, reordering, instructions, pseudo-answer retrieval) for VQA with Open-Flamingo v1/v2 and IDEFICS, and reports three inner properties of LVLM ICL.
Problem
Random sampling of in-context demonstrations for large vision-language models yields sub-optimal VQA performance, and little is known about how demonstration configuration affects LVLM ICL or its internal properties.
Approach
Extends the NLP task recognition (TR) vs task learning (TL) hypothesis to LVLMs by splitting TR into format, visual and linguistic components, and uses diverse retrieval and manipulation strategies on VQA to both improve ICL and probe LVLM behavior.
Key results
- Similar-image retrieval (SI) consistently improves over random sampling, e.g., +14.23 points on VizWiz for OFv2 at 4-shot.
- Similar question-answer retrieval (SQA, an oracle using ground-truth answers) reaches 65.71 on VQAv2 for OFv1 versus 45.97 for RS.
- Similar-question retrieval (SQ) produces copy rates of 77.26%/79.84%/75.34% for OFv1/OFv2/IDEFICS, indicating a short-cut effect, and SQPA with pseudo-answers improves 4-shot performance on OK-VQA (34.82 to 38.92 at 4-shot for OFv2 RS vs SQPA).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| retrieve demonstrations by similar images instead of random sampling | Open-Flamingo v2 · VQAv2, VizWiz, and OK-VQA | accuracy: 1.54/14.23/2.66 point improvements | Helped |
| add instructions to random-sampled demonstrations | Open-Flamingo v2 · VizWiz | accuracy: 22.07 -> 25.70 | Helped |
| add instructions to random-sampled demonstrations | Open-Flamingo v2 · OK-VQA | accuracy: 34.82 -> 36.45 | Helped |
| add instructions to random-sampled demonstrations | Open-Flamingo v1 · VQAv2 | accuracy: 44.56 -> 43.75; 47.38 -> 46.91; 48.71 -> 48.67 | No clear effect |
| retrieve demonstrations using pseudo answers from RS or SI | Open-Flamingo v2 · VQAv2, VizWiz, and OK-VQA, 4-shot | accuracy: 48.82 -> 49.85; 22.07 -> 30.02; 34.82 -> 38.92; 50.36 -> 50.57; 36.30 -> 38.37; 36.46 -> 39.34 | Helped |
| retrieve demonstrations using pseudo answers from RS or SI | Open-Flamingo v2 · VQAv2, 8-shot and 16-shot | accuracy: 51.05 -> 51.03; 50.89 -> 51.96; 32.35 -> 31.93; 39.01 -> 34.25; 38.54 -> 41.16; 39.55 -> 40.06; 52.95 -> 52.02; 54.10 -> | Mixed |
| replace demonstration images with mismatched images (ablation) | Open-Flamingo · VQAv2, VizWiz, and OK-VQA | accuracy: 45.97 -> 45.13; 27.00 -> 26.92; 36.13 -> 35.96; 49.94 -> 49.73; 27.21 -> 26.92; 36.68 -> 36.16 | No clear effect |
| replace demonstration answers with mismatched answers (ablation) | Open-Flamingo · VQAv2, VizWiz, and OK-VQA | accuracy: 45.97 -> 45.65; 27.00 -> 12.77; 36.13 -> 35.56; 49.94 -> 48.94; 27.21 -> 11.51; 36.68 -> 29.88 | Hurt |
| replace question-answer pairs with mismatched pairs (ablation) | Open-Flamingo · VQAv2, VizWiz, and OK-VQA | accuracy: 48.48 -> 47.76; 38.21 -> 27.57; 39.87 -> 37.37; 51.66 -> 49.94; 39.10 -> 26.73; 38.35 -> 35.82 | Hurt |
| retrieve demonstrations by similar questions rather than random sampling | Open-Flamingo · VQAv2 | accuracy: The paper reports that SQ performs worse than RS; no exact paired values are given in the text. | Hurt |
| retrieve demonstrations by similar images and reorder them by question similarity | Open-Flamingo v1 and v2 · VQAv2, VizWiz, and OK-VQA | accuracy | Helped |
Surprising findings
- Mismatching image-question-answer triplets degrades VQA accuracy by less than 1 point on VQAv2, suggesting task recognition matters more than task learning.
- Retrieving demonstrations with similar questions often hurts performance relative to random sampling due to a copy-based short-cut effect.
Limitations
- Analysis is restricted to Open-Flamingo and IDEFICS; generalization to other LVLMs is untested.
- SQA uses ground-truth answers and is not practical at inference; SQPA gains diminish with more shots except on OK-VQA.
Experimental setup
Datasets: VQAv2 VizWiz OK-VQA
Benchmarks: VQAv2 VizWiz OK-VQA
Metrics: VQA accuracy copy rate
Baselines: Random sampling (RS) Similar image (SI) Similar question (SQ) Similar question-answer (SQA) IDEFICS
Builds on: Task recognition and task learning decomposition (Min et al.) Flamingo Open-Flamingo CLIP
Closest papers by meaning
- LIVE: Learnable In-Context Vector for Visual Question Answering (NeurIPS 2024 · similarity 0.92)
- VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.91)
- Lever LM: Configuring In-Context Sequence to Lever Large Vision Language Models (NeurIPS 2024 · similarity 0.89)
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.89)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.89)
- FRAMES-VQA: Benchmarking Fine-Tuning Robustness across Multi-Modal Shifts in Visual Question Answering (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.