Research map / Large language models / Language, safety and interpretability

Making Text Embedders Few-Shot Learners

ICLR 2025 · Large language models › Language, safety and interpretability · approach: In-context learning analysis · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes bge-en-icl, which puts few-shot task examples into the query side of a Mistral-7B LLM embedder trained with contrastive loss, reaching SOTA on MTEB and AIR-Bench.

Problem

LLM-based text embedders generalize poorly to unseen task instructions and complex retrieval tasks; the paper asks whether in-context learning can make embeddings more adaptable.

Approach

Few-shot query-passage examples are prepended to the query during contrastive fine-tuning (with a randomized 0-5 example sampling from in-batch pairs), so the LLM's ICL ability is used to generate task-adapted embeddings, while keeping the original causal attention and last-token [EOS] pooling.

Key results

• bge-en-icl few-shot reaches 71.67 average on MTEB (56 datasets, full data), vs 71.19 for stella_en_1.5B_v5, the prior best listed.
• With public data only, few-shot bge-en-icl scores 66.08 on MTEB vs 66.08 vs 64.80 for LLM2Vec-Mistral-supervised, a gain of 1.25 over its no-ICL variant.
• On AIR-Bench QA the few-shot model averages 54.36 (full data) and 55.92 (public data), above zero-shot by about 1.4 points.

Measured effects reported in the paper

Surprising findings

• Using fixed in-context examples during training substantially hurts zero-shot performance (61.50 vs 64.83 without ICL), while in-batch sampled examples preserve it.
• Bidirectional attention and mean pooling give no significant gain; the original causal attention with last-token pooling works best in the ICL setting.

Limitations

• Full-data results may be confounded by overlap with MTEB-related classification and clustering training data, making comparisons unfair across models with undisclosed data.
• ICL is not explored for the multilingual bge-multilingual-gemma2 model.
• Few-shot examples are limited to 2048-token query length and up to 5 examples, constraining scalability.

Experimental setup

Datasets: ELI5 HotpotQA FEVER MSMARCO NQ NLI SQuAD TriviaQA Quora Duplicate Questions MIRACL Mr.TyDi DuReader

Benchmarks: MTEB AIR-Bench MIRACL FR-MTEB PL-MTEB C-MTEB

Metrics: nDCG@10 Recall@10 Recall@100 average score across MTEB tasks

Baselines: E5-mistral-7b-instruct GritLM-7B SFR-Embedding Linq-Embed-Mistral voyage-large-2-instruct NV-Embed-v1 gte-Qwen2-7B-instruct stella_en_1.5B_v5 LLM2Vec-Mistral-supervised text-embedding-3-large

Builds on: E5-mistral LLM2Vec GritLM NV-Embed InfoNCE

Closest papers by meaning

• Vector-ICL: In-context Learning with Continuous Vector Representations (ICLR 2025 · similarity 0.90)
• A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation (CVPR 2025 · similarity 0.89)
• Can LLMs Reason Over Non-Text Modalities in a Training-Free Manner? A Case Study with In-Context Representation Learning (NeurIPS 2025 · similarity 0.89)
• In-Context Language Learning: Architectures and Algorithms (ICML 2024 · similarity 0.89)
• Revisiting In-context Learning Inference Circuit in Large Language Models (ICLR 2025 · similarity 0.88)
• In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering (ICML 2024 · similarity 0.88)