Research map / Vision-language and multimodal / Multimodal LLMs

ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ARK, a dual-axis (knowledge domain × reasoning skill) multimodal retrieval benchmark with hard negatives, and evaluates 25 text, multimodal, and agentic retrievers, finding large gaps on reasoning-intensive and vision-centric queries.

Problem

Existing multimodal retrieval benchmarks emphasize semantic matching on daily-life images and do not separately diagnose professional knowledge versus multi-step reasoning in retrieval.

Approach

A benchmark that annotates each retrieval instance along two orthogonal axes (five knowledge domains with 17 subtypes, and six reasoning skills plus cognitive demand), covers 16 visual data types, and pairs most queries with targeted hard negatives to block shortcut matching.

Key results

• Best embedding model Seed1.6-embedding reaches 19.32 macro-average Recall@1 on ARK, versus 75.59 R@1 on MMEB-v2.
• Query rewriting with GPT-5.2 raises Seed1.6 R@1 from 19.32 to 23.42 on the knowledge axis.
• Perception-only instances are easiest; fine-grained visual and spatial reasoning remain the hardest skills across models.

Surprising findings

• Text retrievers on captions are competitive on some subtypes, but reasoning-oriented text retrievers (ReasonIR, Diver) do not consistently dominate on ARK.
• Scaling model size improves retrieval but does not eliminate reasoning bottlenecks such as fine-grained visual and spatial reasoning.

Limitations

• Benchmark is modest in size (1,750 queries) because each instance is manually curated.
• Text retrievers depend on captions generated by Qwen3-VL-8B, which can discard visual evidence.
• Hard negatives are not constructed for all queries ("whenever feasible").

Experimental setup

Datasets: COCO Fashion200K DIV8K MindCube Spatial-DISE-12K INQUIRE EMMA MR^2-Bench ReMI Bulletin Editorial Cartoons SemArt arXiv

Benchmarks: ARK

Metrics: Recall@1 Recall@5 Recall@10 nDCG@5 nDCG@10 nDCG@20

Baselines: BGE-M3 Diver-Embed Qwen3-Embed ReasonIR CLIP BGE-VL SigLIP2 MetaCLIP2 VLM2Vec-V2 Ops-MM-Embed Qwen3-VL-Embed LamRA

Builds on: MR^2-Bench MRMR MindCube Spatial-DISE ReMI INQUIRE EMMA

Closest papers by meaning

• RMIR: A Benchmark Dataset for Reasoning-Intensive Multimodal Image Retrieval (CVPR 2026 · similarity 0.94)
• MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.91)
• MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval (ICLR 2026 · similarity 0.91)
• Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.91)
• Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph (ICML 2026 · similarity 0.91)
• MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents (ICLR 2026 · similarity 0.90)