Research map / Multi-agent systems and games / LLM agents
AOEB: Benchmarking Agent-Oriented Multimodal Embeddings
Xin Zhang, Jiaxin Xu, mengjia zhou, Xinping Zhao, Yinghui Li, di yin, Xing Sun, Meishan Zhang, Baotian Hu, Wenjie Li, Min Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces AOEB, a benchmark for evaluating multimodal embedding models on agent-oriented retrieval scenarios, motivated by gaps in general-purpose embedding benchmarks.
Problem
Current embedding benchmarks focus on general-purpose scenarios and may not match the diverse, evolving retrieval needs of LLM agent applications such as RAG.
Approach
A benchmark specifically oriented to agentic applications for multimodal embeddings, rather than general-purpose embedding evaluation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| render interleaved content as a single document image (Visual-doc) instead of merging images (ablation) | Jina-v4 on interleaved datasets · MRMR traffic | nDCG@10: 15.33 -> 25.14 | Improved |
| use Visual-doc instead of Merge-image (ablation) | VISTA · MRMR traffic | nDCG@10: 21.53 -> 6.69 | Hurt |
| use Visual-doc instead of Merge-image (ablation) | VLM2Vec · MMRC-R session | nDCG@10: 46.39 -> 68.91 | Improved |
| Visual-doc vs Merge-image across models (average diff) (ablation) | single-image embedding models · AOEB interleaved datasets | nDCG@10 diff: VISTA -25.6, mmE5 +11.1, VLM2Vec-V2 +8.0, Qwen3VL2B -1.0 | Mixed |
| choose the model with stronger AOEB-T macro average (GME7B) vs micro (Qwen38B) | ranking metric · AOEB-T | Avg.Type vs Avg.43: GME7B 39.67 vs Qwen38B 38.86 macro; 34.23 vs 34.45 micro | Mixed |
| better AOEB-T embedding model used in downstream RAG | FreshStack RAG · FreshStack end-to-end | RAG coverage: 0.553 (E5Mistral) / 0.584 (Jina-v4) -> 0.689 (Qwen38B) | Improved |
| better embedding model used in deep research agent | BrowseComp-Plus agent · BrowseComp-Plus | accuracy: 3.13 (Jina-v4) / 4.78 (E5Mistral) -> 8.55 (Qwen38B) | Improved |
| multimodal MLLM-based embedder vs text-only embedder at same scale | text-only retrieval · AOEB-T | Avg.Type: Qwen38B 38.86 vs GME7B 39.67 macro | No clear effect |
| general-embedding-trained models (mmE5, VLM2Vec) on agent tasks | mmE5 and VLM2Vec · AOEB | Avg.Type: mmE5 39.97, VLM2Vec 26.12 vs Qwen3VL2B 51.03 | Hurt |
| frozen text embedding space encoder (VISTA) vs MLLM-based embedders | VISTA · AOEB | Avg.Type: 30.58 vs 51.03 (Qwen3VL2B) | Hurt |
| better performance on general benchmarks (MTEB, RTEB, MMEB) | embedding models · AOEB | rank | No clear effect |
Experimental setup
Benchmarks: AOEB
Closest papers by meaning
- Fix Before Search: Benchmarking Agentic Visual Query Pre-processing in Multimodal Retrieval-augmented Generation (ICML 2026 · similarity 0.88)
- MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents (ICLR 2026 · similarity 0.87)
- MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction (ICLR 2026 · similarity 0.87)
- Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph (ICML 2026 · similarity 0.87)
- OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents (NeurIPS 2026 · similarity 0.87)
- MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
