Research map / Vision-language and multimodal / Multimodal LLMs

ContextNav: Towards Agentic Multimodal In-Context Learning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ContextNav, an agentic framework that automates multimodal in-context example retrieval, denoises candidates (semantic and structural), and optimizes tool workflows via a graph and memory of ICL feedback.

Problem

Multimodal ICL with retrieved demonstrations is sensitive to semantically and structurally noisy contexts, while manual curation does not scale; static one-shot retrieval cannot adapt from downstream feedback.

Approach

An MLLM agent builds a resource-aware multimodal vector database, retrieves top-k candidates, filters them with agentic retrieval (semantic check), rewrites them to match query structure, and plans toolchains over an Operational Grammar Graph, refining plans using memory of downstream ICL feedback.

Key results

• ContextNav with Gemini-2.0-flash reaches average accuracy 0.604 vs 0.542 for vanilla and 0.561 for VL-ICL across the eight datasets.
• Average ICL gain of 16.8% across models versus prior state-of-the-art 7.6% (as stated by the authors).
• Ablation: removing agentic retrieval drops ICL gain from +11.8% to +1.6% on MathVision; removing textual retrieval tools and AR yields -18.7%.

Surprising findings

• Random sampling of demonstrations often degrades performance below the zero-shot baseline.
• Qwen3-Embedding-8B and 4B give nearly identical retrieval, suggesting larger embedding models bring diminishing returns.

Limitations

• Extra token overhead (about 22.5K tokens per iteration) and latency limit real-time use.
• Final predictions still limited by downstream MLLM reasoning, as shown in failure cases.
• Gains on some datasets (e.g., FOMI, RealWorld for some models) are small or zero.

Experimental setup

Datasets: BlindTest MME-RealWorld CharXiv GVL MathVision CLEVR FOMI TextOCR

Benchmarks: BlindTest MME-RealWorld CharXiv GVL MathVision CLEVR FOMI TextOCR

Metrics: accuracy ICL gain (%) semantic noise ratio structural noise ratio toolchain success rate

Baselines: Random sampling VL-ICL MMICES vanilla MLLMs

Builds on: VL-ICL MMICES

Closest papers by meaning

• VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning (ICLR 2025 · similarity 0.90)
• AIM: Let Any Multimodal Large Language Models Embrace Efficient In-Context Learning (AAAI 2025 · similarity 0.89)
• Improving Context Understanding in Multimodal Large Language Models via Multimodal Composition Learning (ICML 2024 · similarity 0.89)
• Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.89)
• In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering (ICML 2024 · similarity 0.89)
• ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models (ICML 2024 · similarity 0.88)