Research map / Vision-language and multimodal / Multimodal LLMs

MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MetaEmbed, which appends learnable Meta Tokens to VLM inputs and trains nested coarse-to-fine multi-vector embeddings (Matryoshka Multi-Vector Retrieval) so retrieval budget can be chosen at test time.

Problem

Single-vector multimodal embeddings lose fine-grained information, while ColBERT-style multi-vector methods have prohibitive index size, latency, and cost, especially for multimodal-to-multimodal retrieval.

Approach

Learnable Meta Tokens appended to query and candidate inputs yield a small set of contextualized multi-vector embeddings, trained with prefix-nested groups (Matryoshka-style) in parallel contrastive losses so that the number of vectors used at indexing and query time can be traded off against accuracy without retraining.

Key results

• MetaEmbed-7B reaches 76.6 overall Precision@1 on MMEB at (16,64) budget, versus 71.5 for MoCa-7B and 69.8 for mmE5.
• MetaEmbed-32B reaches 78.7 overall on MMEB, the reported state of the art in the paper.
• On ViDoRe v2, MetaEmbed-7B averages 61.3 NDCG@5 versus 59.3 for GME-7B and 57.5 for ColQwen2.

Surprising findings

• Without MMR, the model gains only marginally at full budget (NoMMR 69.3 vs 69.1 for 3B on MMEB), but MMR is far better at low budgets, dropping 9.0 points at (1,1) without it on ViDoRe v1.
• The single-vector baseline's gain from 7B to 32B is not statistically significant, while MetaEmbed still improves.

Limitations

• Index memory grows proportionally with retrieval budget, reaching 42.72 GiB at (16,64) for 100k candidates.
• Performance of the Llama-3.2-Vision-based variant is weak on VQA, inherited from the backbone's generative weaknesses.
• Multimodal-to-multimodal retrieval and training data are limited to MMEB and ViDoRe training sets; no explicit multilingual training data.

Experimental setup

Datasets: MMEB-train ViDoRe-train

Benchmarks: MMEB ViDoRe v2 ViDoRe v1

Metrics: Precision@1 NDCG@5 latency index memory scoring FLOPs

Baselines: CLIP MagicLens UniIR MM-Embed GME VLM2Vec VLM2Vec-V2 MMRet mmE5 MoCa B3 SigLIP

Builds on: ColBERT Matryoshka Representation Learning VLM2Vec LoRA

Closest papers by meaning

• MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS (ICLR 2025 · similarity 0.92)
• Your Embedding Model Is SMARTer Than You Think (NeurIPS 2026 · similarity 0.92)
• U-MARVEL: Unveiling Key Factors for Universal Multimodal Retrieval via Embedding Learning with MLLMs (ICLR 2026 · similarity 0.91)
• NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.89)
• VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.88)
• Beyond Global Similarity: Multi-Conditional Retrieval for Fine-Grained Cross-Modal Understanding (CVPR 2026 · similarity 0.88)