Research map / Vision-language and multimodal / Visual question answering
Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Visual Haystacks, a vision-centric needle-in-a-haystack benchmark for multi-image QA over up to 10K COCO images, and MIRAGE, a visual-RAG LMM with a Q-Former compressor and learned retriever that scales to 10K images on one 40GB A100.
Problem
Long-context LMMs can ingest many images, but they fail at retrieving and reasoning over relevant visual content among distractors in multi-image QA; existing visual needle-in-a-haystack benchmarks overlay text and thus mostly test OCR.
Approach
A realistic, vision-centric NIAH benchmark built from COCO with real visual needles (anchor object retrieval then target object reasoning), plus MIRAGE, which compresses each image to 32 tokens via a Q-Former and filters images with a query-aware retriever trained jointly with next-token prediction on LLaVA-based visual RAG.
Key results
- MIRAGE-8.3B reaches 67.6 on RetVQA, setting a new state of the art on that benchmark and beating GPT-4o (34.6) and Gemini v1.5 Pro (32.2) in the reported table.
- On single-needle VHs, Gemini 1.5 Pro scores 88.35% at N=1 and falls to 57.42% at N=100, while MIRAGE-8.3B keeps 61.99% at N=100 and reaches 51.70% at N=10K.
- MIRAGE-8.3B is reported to run on a single 40GB A100 GPU and handle up to 10k images, versus about 1k-image limits for contemporary models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MIRAGE-8.3B instead of the strongest listed baseline | RetVQA · RetVQA | score: 30.6 -> 67.6 | Helped |
| Use MIRAGE-8.3B instead of LLaVA-v1.5-7B | single-image QA · VQAv2 | score: 78.5 -> 76.6 | Hurt |
| Use MIRAGE-8.3B instead of LLaVA-v1.5-7B | single-image QA · GQA | score: 62.0 -> 59.1 | Hurt |
| Use MIRAGE-8.3B instead of LLaVA-v1.5-7B | single-image QA · TextVQA | score: 58.2 -> 56.2 | Hurt |
| Use MIRAGE-8.3B instead of LLaVA-v1.5-7B | single-image QA · MMB | score: 64.3 -> 69.2 | Helped |
| Use MIRAGE-8.3B instead of LLaVA-v1.5-7B | single-image QA · MMB-CN | score: 58.3 -> 66.9 | Helped |
| Use MIRAGE-8.3B instead of LLaVA-v1.5-7B | single-image QA · MM-Vet | score: 31.1 -> 33.4 | Helped |
| Increase the haystack from N=1 to N=100 | MIRAGE-8.3B · VHs single-needle | accuracy: 83.24%±1.11% -> 61.99%±1.41% | Hurt |
| Increase the haystack from N=1K to N=10K | MIRAGE-8.3B · VHs single-needle | accuracy: 55.70%±1.57% -> 51.70%±2.56% | Hurt |
| Use MIRAGE-8.3B instead of CLIP + LLaVA-v1.5-7B | VHs single-needle · N=10K | accuracy: 49.30%±2.20% -> 51.70%±2.56% | Helped |
| Increase the number of needle images from one to two | LMMs · VHs oracle cross-image reasoning | accuracy | Hurt |
| Place the needle at an unfavorable position in the input image sequence | existing LMMs · VHs positional evaluation | accuracy: up to 25% | Hurt |
Surprising findings
- GPT-4o exhibits a lost-in-the-middle positional effect, while Gemini v1.5 Pro favors needles at the beginning of the sequence.
- Caption aggregation with a text-only LLM matches or surpasses open-source LMMs once the haystack reaches about 20 images.
Limitations
- Benchmark is template-based on COCO's 80 object categories, limiting generalization and language reasoning evaluation.
- MIRAGE sacrifices some single-image QA performance relative to LLaVA-v1.5-7B due to token compression.
- MIRAGE lags Gemini v1.5 Pro on multi-needle tasks and drops from single- to multi-needle settings.
Experimental setup
Datasets: RetVQA SlideVQA WebQA LLaVA instruction data COCO CC3M/12M LAION-400M ShareGPT captioning
Benchmarks: Visual Haystacks RetVQA VQAv2 GQA TextVQA POPE MMBench MME SEED-Bench MM-Vet
Metrics: accuracy recall
Baselines: GPT-4o Gemini v1.5 Pro Qwen2-VL-7B InternVL2-8B Phi-3 Vision mPLUG-Owl3 CLIP + LLaVA-v1.5-7B LWM
Builds on: LLaVA-v1.5 Q-Former retrieval-augmented generation needle-in-a-haystack
Closest papers by meaning
- Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · similarity 0.94)
- MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.90)
- BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions (AAAI 2024 · similarity 0.90)
- FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering (NeurIPS 2025 · similarity 0.90)
- Re-ranking Reasoning Context with Tree Search Makes Large Vision-Language Models Stronger (ICML 2025 · similarity 0.89)
- Multimodal Hypothetical Summary for Retrieval-based Multi-image Question Answering (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.