Research map / Vision-language and multimodal / Multimodal LLMs
MultiHaystack: Benchmarking Multimodal Retrieval and Reasoning over 40K Images, Videos, and Documents
Dannong Xu, zhongyu yang, Jun Chen, Yingfang Yuan, Ming Hu, Lei Sun, Luc Van Gool, Danda Paudel, Chun-Mei Feng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| move from single-modality to cross-modality candidate pool (ablation) | E5-V · MultiHaystack retrieval (overall) | Recall@1 (%): 50.87 -> 40.83 | Hurt |
| move from single-modality to cross-modality candidate pool (ablation) | SigLIP2 · MultiHaystack retrieval (overall) | Recall@1 (%): 51.81 -> 40.96 | Hurt |
| move from single-modality to cross-modality candidate pool (ablation) | CLIP · MultiHaystack retrieval (overall) | Recall@1 (%): 36.28 -> 26.10 | Hurt |
| enlarge candidate pool from 1K to 46K (ablation) | E5-V · MultiHaystack retrieval | Recall@1 (%): 72.42 -> 40.83 | Hurt |
| provide gold evidence versus top-5 E5-V retrieval (ablation) | GPT-5 · MultiHaystack reasoning | accuracy (%): 80.86 -> 51.4 | Hurt |
| retrieve cross-modality instead of single-modality (ablation) | GPT-5 · MultiHaystack reasoning (E5-V top-5) | accuracy (%): 59.84 -> 51.41 | Hurt |
| retrieve cross-modality instead of single-modality (ablation) | Gemini-2.5-Flash · MultiHaystack reasoning (E5-V top-5) | accuracy (%): 50.87 -> 43.64 | Hurt |
| add query refinement to E5-V (ablation) | E5-V · MultiHaystack retrieval (cross-modality) | Recall@1 (%): 40.83 -> 43.78 | Improved |
| add MMSearch pipeline to E5-V (ablation) | E5-V · MultiHaystack retrieval (cross-modality) | Recall@1 (%): 40.83 -> 44.58 | Improved |
| use VisRAG instead of E5-V | E5-V · MultiHaystack retrieval (cross-modality) | Recall@1 (%): 40.83 -> 45.38 | Mixed |
| increase retrieved items from top-1 to top-5 (ablation) | GPT-5 with E5-V · MultiHaystack reasoning | accuracy (%): 64.11 at Top-5 | Improved |
| mix modalities in the pool at same total size versus single-modality pool (ablation) | MM-Embed · MultiHaystack retrieval | Recall | Hurt |
Closest papers by meaning
- ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge (NeurIPS 2026 · similarity 0.92)
- Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.92)
- RMIR: A Benchmark Dataset for Reasoning-Intensive Multimodal Image Retrieval (CVPR 2026 · similarity 0.92)
- MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval (ICLR 2026 · similarity 0.91)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)
- Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
