Research map / Vision-language and multimodal / Multimodal LLMs
Identifying and Mitigating Position Bias of Multi-image Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies position bias in multi-image LVLMs via a position-wise QA task and proposes SoFA, a training-free soft interpolation between inter-image causal and bidirectional attention that reduces the bias and modestly improves accuracy.
Problem
Multi-image vision-language models are sensitive to the order of input images, showing position-dependent reasoning accuracy (recency bias in open-source models, middle-position weakness in proprietary ones).
Approach
Introduce Position-wise Question Answering to measure per-position reasoning, attribute position bias to inter-image causal attention, and mitigate it by linearly interpolating the inter-image attention mask between causal and bidirectional masks (SoFA), applied every two layers without training.
Key results
- SoFA raises Idefics2 BLINK prediction inconsistency from 41.55% to 12.36% across image orderings.
- SoFA improves InternVL2 BLINK accuracy from 38.95% to 43.26%, and Idefics2 MIRB from 46.58% to 48.83%.
- On Visual Haystack with 100 images, SoFA raises average accuracy from 49.19% to 55.11%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SoFA (linear interpolation of inter-image causal and bidirectional attention) | Idefics2-8B · BLINK | prediction inconsistency (lower is better): 41.55 -> 12.36 | Helped |
| Add SoFA | Multiple LVLMs (Idefics2, InternVL2, VILA, Mantis, LLaVA-NeXT) · Position-agnostic tasks (BLINK, Mantis-Eval, MuirBench, MIRB) | prediction inconsistency (lower is better): Reduced inconsistency in all 20 model-benchmark pairs; e.g. InternVL2 MuirBench 38.65 -> 5.16 | Helped |
| Add SoFA | Five open-source LVLMs · Overall benchmarks (BLINK, Mantis-Eval, MuirBench, MIRB, NLVR2, MVBench) | accuracy: Gains on most model-benchmark pairs (e.g. Idefics2 BLINK +2.53, InternVL2 BLINK +4.31) but Mantis NLVR2 90.26 -> 89.98 d | Mixed |
| Add SoFA | LLaVA-NeXT-Interleave-7B · MVBench | accuracy: 54.26 -> 57.71 | Helped |
| Add SoFA | Idefics2 and VILA long-context and ICL settings · Multi-modal Haystack (100 images) | accuracy: 49.19 -> 55.11 | Helped |
| Add SoFA | LVLMs in multi-shot in-context learning · VizWiz 16-shot | accuracy: 45.35 -> 49.17 | Helped |
| Set interpolation weight sigma to 0 (pure inter-image causal attention) vs tuned sigma (ablation) | SoFA on BLINK · BLINK | accuracy: 47.85 at sigma 0 vs 50.04 at sigma 0.50 | Hurt |
| Set interpolation weight sigma to 1.0 (pure bidirectional) vs tuned sigma (ablation) | SoFA on MVBench · MVBench | accuracy: 34.20 at sigma 1.00 vs 37.28 at sigma 0.25 | Hurt |
| Replace causal inter-image attention with isolated or bidirectional attention (no interpolation) (ablation) | LVLMs on PQA · PQA | position-wise accuracy balance and overall accuracy | Mixed |
| Add SoFA to multi-shot ICL with more demonstrations | LVLMs on VQAv2, VizWiz, OK-VQA ICL · VQAv2 4-shot to 16-shot | accuracy: 55.34 -> 56.13 (4-shot), 61.37 -> 64.85 (16-shot) | Helped |
Surprising findings
- Proprietary GPT-4o shows a U-shaped position curve, struggling with middle-positioned images, while open-source models show recency bias.
- Isolated attention, one of the position-bias-reducing alternatives, causes the largest performance drop, likely due to out-of-distribution hidden states.
Limitations
- Requires a small validation set per task to tune the interpolation weight sigma, which varies across benchmarks.
- Evaluation relies on GPT-4 as judge for correctness.
- Gains are modest (about 2-3% average) and mainly tested on position-agnostic tasks.
Experimental setup
Datasets: VQAv2 BLINK Mantis-Eval MuirBench MIRB NLVR2 MVBench VizWiz OK-VQA Visual Haystack
Benchmarks: BLINK Mantis-Eval MuirBench MIRB NLVR2 MVBench VQAv2 VizWiz OK-VQA Visual Haystack PQA
Metrics: accuracy min/avg/max accuracy across image shuffles prediction inconsistency
Baselines: Idefics2 InternVL2 VILA Mantis LLaVA-NeXT-Interleave OpenFlamingo GPT-4o
Builds on: lost in the middle LLaVA-NeXT-Interleave VQAv2
Closest papers by meaning
- Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ICML 2025 · similarity 0.89)
- Unveiling the Tapestry of Consistency in Large Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Enhancing Large Vision Language Models with Self-Training on Image Comprehension (NeurIPS 2024 · similarity 0.89)
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.89)
- Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs? (ICML 2025 · similarity 0.89)
- Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.