Research map / Vision-language and multimodal / CLIP and zero-shot

MMTEB: Massive Multilingual Text Embedding Benchmark

ICLR 2025 · Vision-language and multimodal › CLIP and zero-shot · approach: Language models for zero-shot reasoning · 16 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MMTEB, a community-built benchmark of 500+ quality-controlled multilingual embedding tasks, plus efficient downsampling and task-selection methods to create smaller benchmark variants and evaluates a range of embedding models.

Problem

Text embedding models are evaluated on narrow sets of tasks, languages, and domains, and full evaluation of large embedding models is computationally expensive, which disadvantages low-resource language communities.

Approach

Crowd-sourced expansion of MTEB to 500+ tasks across 250+ languages, with downsampling (TREC-style pooling for retrieval, bootstrapped clustering with cached embeddings) and correlation-based task selection to build cheaper benchmarks that preserve model rankings.

Key results

• multilingual-e5-large-instruct (560M) ranks first on MTEB(Multilingual) with mean score 63.2, the best publicly available model in the study.
• MTEB(eng, v2) with 41 tasks correlates with MTEB(eng, v1) at Spearman 0.90 and runs in 3.11 hours for GritLM-7B on an H100 while using about 2% of original documents.
• Retrieval downsampling preserves rankings; the English zero-shot benchmark keeps similar ordering using 2% of documents.

Measured effects reported in the paper

Surprising findings

• Smaller XLM-R-based multilingual-e5-large(-instruct) outperforms 7B Mistral-based GritLM-7B and e5-mistral-7b-instruct on multilingual and lower-resource benchmarks.
• English-only small models like all-MiniLM perform worse on zero-shot MTEB(eng, v2), likely because MS MARCO and Natural Questions were removed from training-leaky tasks.

Limitations

• Skew toward high-resource languages in task distribution despite 250+ languages covered.
• English leakage: human translations of English datasets may bias toward English content.
• Credit assignment via point system is imperfect, with equal points for datasets of differing effort.

Experimental setup

Datasets: MS MARCO Natural Questions FollowIR Flores SIB200

Benchmarks: MTEB(Multilingual) MTEB(Europe) MTEB(Indic) MTEB(eng, v2) MTEB(eng, v1)

Metrics: Borda count rank average score Spearman correlation nDCG@10 v-measure accuracy F1

Baselines: multilingual-e5-large-instruct GritLM-7B e5-mistral-7b-instruct multilingual-e5-large LaBSE multilingual-mpnet-base all-MiniLM-L12

Builds on: MTEB BEIR multilingual-e5 TREC pooling

Closest papers by meaning

• The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding (NeurIPS 2024 · similarity 0.95)
• Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.90)
• NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR 2025 · similarity 0.88)
• EmbedLLM: Learning Compact Representations of Large Language Models (ICLR 2025 · similarity 0.88)
• Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.88)
• Making Text Embedders Few-Shot Learners (ICLR 2025 · similarity 0.87)