Research map / Vision-language and multimodal / Multimodal LLMs

UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes UME-R1, a two-stage (CoT-augmented SFT then GRPO with ranking-and-similarity-gap embedding reward) framework that lets MLLM embedders produce either discriminative or reasoning-driven generative embeddings, reaching 64.5 overall on MMEB-V2.

Problem

Existing MLLM-based multimodal embedding models are discriminative (last-token hidden state without generating tokens), so they cannot benefit from reasoning-driven generation; RLVR has no standard answer for embedding tasks.

Approach

Generate reasoning and a summary before the embedding token so the embedding is a reasoning-driven generative representation, trained with contrastive loss on embedding tokens plus next-token loss on reasoning, then refined with GRPO using a reward combining top-G ranking of positives and the similarity gap between positive and negative targets.

Key results

• UME-R1-7B reaches 64.5 overall on MMEB-V2 (78 tasks), above VLM2Vec-V2 (58.0) and DUME-7B (55.9) trained on the same data.
• Oracle selection between generative and discriminative embeddings raises UME-R1-7B to 68.1 overall and UME-R1-2B to 64.4.
• Ablations show RL adds about 1.0 point over SFT-only UME for UME-R1-2B (60.1 vs 59.1), and removing either the ranking or similarity-gap reward lowers performance.

Surprising findings

• Reasoning-driven generative embeddings from the model's own reasoning outperform feeding reasoning from a stronger external 9B model (GLM-4.1V-Thinking) into a discriminative embedder (DUME+Gen).
• The 2B model's completion length decreases during RL, which the authors attribute to limited reasoning capacity making long reasoning harmful.

Limitations

• Reasoning-driven generative embeddings incur substantially higher training and inference cost due to long CoT generation.
• No practical router exists to choose between discriminative and generative embeddings; the oracle is an upper bound only.
• RL dataset is small (about 11K pairs) and the fixed-threshold reward variant gives limited gains outside video.

Experimental setup

Datasets: MMEB-V2 training data MMEB LLaVA-Hound ViDoRe VisRAG OK-VQA ChartQA CIRR A-OKVQA Visual7W

Benchmarks: MMEB-V2 MMEB-V1

Metrics: Hit@1 NDCG@5 pass@k

Baselines: GME ColPali VLM2Vec LamRA CAFe VLM2Vec-V2 DUME

Builds on: VLM2Vec GRPO DeepSeek-R1 VLM2Vec-V2 DUME

Closest papers by meaning

• Think Then Embed: Generative Context Improves Multimodal Embedding (ICLR 2026 · similarity 0.93)
• VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.90)
• Scaling Language-centric Omnimodal Representation Learning (NeurIPS 2025 · similarity 0.89)
• Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
• Unified Generative and Discriminative Training for Multi-modal Large Language Models (NeurIPS 2024 · similarity 0.88)
• DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models (ICML 2026 · similarity 0.88)