Research map / Vision-language and multimodal / Multimodal LLMs
Needle In A Multimodal Haystack
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MM-NIAH, a needle-in-a-haystack benchmark that inserts text and image needles into long interleaved OBELICS image-text documents (1k–72k tokens) to test MLLM retrieval, counting, and reasoning, and finds large gaps especially on image needles.
Problem
Existing multimodal benchmards use short contexts or vision-dominant video, so MLLMs' ability to comprehend long interleaved multimodal documents is underexplored.
Approach
Extends text needle-in-a-haystack to multimodal documents by concatenating OBELICS interleaved image-text sequences as haystacks and inserting text or cartoon-image needles at controlled depths and context lengths, with retrieval, counting, and reasoning tasks; also builds an InternVL-1.5 + RAG baseline.
Key results
- InternVL-1.5 with RAG improves text retrieval from 89.0 to 97.1 overall, but image retrieval only moves from 28.0 to 26.0.
- Gemini-1.5 reaches 22.4 overall on image retrieval and 13.8 on image counting versus 98.1 and 98.2 for human experts.
- Open-source models such as InternVL-1.5 drop from 59.5 at 1K to 30.1 at 64K on overall-average accuracy, and LLaVA-1.6-34B falls to 0 beyond 48K.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add RAG to InternVL-1.5 | InternVL-1.5 · Retrieval-Text-Needle | accuracy: 89.0 -> 97.1 | Helped |
| Add RAG to InternVL-1.5 | InternVL-1.5 · Counting-Text-Needle | soft accuracy: 29.8 -> 45.9 | Helped |
| Add RAG to InternVL-1.5 | InternVL-1.5 · Reasoning-Text-Needle | accuracy: 52.2 -> 69.3 | Helped |
| Add RAG to InternVL-1.5 | InternVL-1.5 · Retrieval-Image-Needle | accuracy: 28.0 -> 26.0 | Hurt |
| Add RAG to InternVL-1.5 | InternVL-1.5 · Counting-Image-Needle | soft accuracy: 5.0 -> 6.7 | Helped |
| Add RAG to InternVL-1.5 | InternVL-1.5 · Reasoning-Image-Needle | accuracy: 50.1 -> 55.3 | Helped |
| Increase multimodal context length | Evaluated MLLMs · MM-NIAH | overall performance: InternVL-1.5: 59.5 -> 30.1 | Hurt |
| Use image needles instead of text needles | InternVL-1.5 · Retrieval | accuracy: 89.0 -> 28.0 | Hurt |
| Use image needles instead of text needles | InternVL-1.5 · Counting | soft accuracy: 29.8 -> 5.0 | Hurt |
| Use image needles instead of text needles | InternVL-1.5 · Reasoning | accuracy: 52.2 -> 50.1 | Hurt |
| Fine-tune InternLM2-20B multimodally using context lengths capped below 4096 | InternLM2-20B · Text retrieval, 32K–72K context | accuracy: 90.7 -> 81.0; 82.7 -> 72.7 | Hurt |
| Place questions before the context | MLLMs · MM-NIAH image-needle tasks | performance | Hurt |
Surprising findings
- Models pre-trained on interleaved image-text data (VILA, Emu2-Chat) do not outperform models trained only on image-text pairs.
- RAG greatly helps text needles but gives negligible gains on image needles, and InternLM2 without images retains long-context ability better than InternVL-1.5.
Limitations
- Long documents serve only as background; questions relate only to inserted needles, not the entire document.
- Context lengths are capped at 72k tokens with at most 36 images.
Experimental setup
Datasets: OBELICS BLINK
Benchmarks: MM-NIAH
Metrics: accuracy soft accuracy
Baselines: LLaVA-1.6 InternVL-1.5 VILA Emu2-Chat IDEFICS2 Gemini-1.5-Flash GPT-4V
Builds on: Needle In A Haystack Counting Stars RULER BABILong
Closest papers by meaning
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.88)
- URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understanding (AAAI 2026 · similarity 0.88)
- Docopilot: Improving Multimodal Models for Document-Level Understanding (CVPR 2025 · similarity 0.88)
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (NeurIPS 2025 · similarity 0.88)
- Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · similarity 0.88)
- NoLiMa: Long-Context Evaluation Beyond Literal Matching (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.