AI Paper Atlas

Research map / Multi-agent systems and games / LLM agents

AOEB: Benchmarking Agent-Oriented Multimodal Embeddings

Xin Zhang, Jiaxin Xu, mengjia zhou, Xinping Zhao, Yinghui Li, di yin, Xing Sun, Meishan Zhang, Baotian Hu, Wenjie Li, Min Zhang

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces AOEB, a benchmark for evaluating multimodal embedding models on agent-oriented retrieval scenarios, motivated by gaps in general-purpose embedding benchmarks.

Problem

Current embedding benchmarks focus on general-purpose scenarios and may not match the diverse, evolving retrieval needs of LLM agent applications such as RAG.

Approach

A benchmark specifically oriented to agentic applications for multimodal embeddings, rather than general-purpose embedding evaluation.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
render interleaved content as a single document image (Visual-doc) instead of merging images (ablation)Jina-v4 on interleaved datasets · MRMR trafficnDCG@10: 15.33 -> 25.14Improved
use Visual-doc instead of Merge-image (ablation)VISTA · MRMR trafficnDCG@10: 21.53 -> 6.69Hurt
use Visual-doc instead of Merge-image (ablation)VLM2Vec · MMRC-R sessionnDCG@10: 46.39 -> 68.91Improved
Visual-doc vs Merge-image across models (average diff) (ablation)single-image embedding models · AOEB interleaved datasetsnDCG@10 diff: VISTA -25.6, mmE5 +11.1, VLM2Vec-V2 +8.0, Qwen3VL2B -1.0Mixed
choose the model with stronger AOEB-T macro average (GME7B) vs micro (Qwen38B)ranking metric · AOEB-TAvg.Type vs Avg.43: GME7B 39.67 vs Qwen38B 38.86 macro; 34.23 vs 34.45 microMixed
better AOEB-T embedding model used in downstream RAGFreshStack RAG · FreshStack end-to-endRAG coverage: 0.553 (E5Mistral) / 0.584 (Jina-v4) -> 0.689 (Qwen38B)Improved
better embedding model used in deep research agentBrowseComp-Plus agent · BrowseComp-Plusaccuracy: 3.13 (Jina-v4) / 4.78 (E5Mistral) -> 8.55 (Qwen38B)Improved
multimodal MLLM-based embedder vs text-only embedder at same scaletext-only retrieval · AOEB-TAvg.Type: Qwen38B 38.86 vs GME7B 39.67 macroNo clear effect
general-embedding-trained models (mmE5, VLM2Vec) on agent tasksmmE5 and VLM2Vec · AOEBAvg.Type: mmE5 39.97, VLM2Vec 26.12 vs Qwen3VL2B 51.03Hurt
frozen text embedding space encoder (VISTA) vs MLLM-based embeddersVISTA · AOEBAvg.Type: 30.58 vs 51.03 (Qwen3VL2B)Hurt
better performance on general benchmarks (MTEB, RTEB, MMEB)embedding models · AOEBrankNo clear effect

Experimental setup

Benchmarks: AOEB

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy