Research map / Large language models / Language, safety and interpretability
NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NV-Embed, a decoder-only Mistral-7B embedding model with latent attention pooling, bidirectional attention, and two-stage contrastive instruction tuning, reaching No.1 on MTEB (72.31 for v2).
Problem
Decoder-only LLMs have underperformed bidirectional encoders on general-purpose text embedding tasks due to causal attention and last-token pooling; the paper seeks a generalist embedding model that excels across retrieval and non-retrieval tasks.
Approach
Remove the causal mask during contrastive training, replace mean/EOS pooling with a trainable latent-attention pooling layer (512 latents), and train in two stages: retrieval with in-batch and hard negatives, then blended retrieval and non-retrieval tasks with in-batch negatives disabled.
Key results
- NV-Embed-v2 achieves a new record average MTEB score of 72.31 across 56 tasks, reclaiming No.1 as of Aug 30, 2024.
- Latent-attention pooling with bidirectional mask gives 72.31 vs 71.63 for EOS and 71.71 for mean pooling in the second-stage ablation of v2.
- On AIR-Bench Long-Doc, NV-Embed-v2 scores 74.78 Recall@10, highest among compared models; in QA it is second with 52.28.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with two stages rather than a single stage with in-batch negatives (ablation) | NV-Embed · MTEB (56 tasks) | Avg.: 70.83 -> 72.31 | Helped |
| train with two stages rather than a single stage without in-batch negatives (ablation) | NV-Embed · MTEB (56 tasks) | Avg.: 71.94 -> 72.31 | Helped |
| reverse the order of the two training stages (ablation) | NV-Embed · MTEB (56 tasks) | Avg.: 72.31 -> 71.85 | Hurt |
| replace label-based class and cluster documents with sampled examples (ablation) | NV-Embed · MTEB (56 tasks) | Avg.: 70.82 -> 72.31 | Helped |
| add hard-negative mining (ablation) | NV-Embed stage-one training · MTEB (56 tasks) | Avg.: 70.73 -> 71.83 | Helped |
| add additional public retrieval datasets (ablation) | NV-Embed stage-one training · MTEB (56 tasks) | Avg.: 71.83 -> 72.07 | Helped |
| add synthetically generated data (ablation) | NV-Embed stage-one training · MTEB (56 tasks) | Avg.: 72.07 -> 72.31 | Helped |
| remove the causal attention mask (ablation) | NV-Embed · MTEB (56 tasks), second-stage training | Average: 68.47 -> 69.32 | Helped |
| use latent-attention pooling instead of mean pooling (ablation) | NV-Embed · MTEB (56 tasks), second-stage training | Average: 68.97 -> 69.32 | Helped |
| use self-attention pooling instead of mean pooling (ablation) | NV-Embed · MTEB (56 tasks), second-stage training | Average: 68.97 -> 69.10 | No clear effect |
| use latent-attention pooling instead of EOS-token pooling (ablation) | NV-Embed · MTEB (56 tasks), second-stage training | Average: 67.85 -> 69.32 | Helped |
| add hard-negative mining (ablation) | NV-Embed stage-one training · MTEB Retrieval (15) | nDCG@10: 59.22 -> 61.52 | Helped |
Surprising findings
- Adding in-batch negatives in single-stage training hurts classification (70.83 avg vs 71.94 without), since in-batch labels can coincide with positives in few-class tasks.
- Adding a self-attention layer before mean pooling gives no meaningful gain over mean pooling, while latent attention does.
Limitations
- Relies on proprietary-generated synthetic data (Mixtral-8x22B) and on E5-mistral-7b-instruct as teacher for hard-negative mining.
- Many training datasets overlap with MTEB training splits, so MTEB gains partly reflect dataset alignment.
- AIR-Bench evaluation is closed-book and only available via HuggingFace Hub, limiting independent verification.
Experimental setup
Datasets: MSMARCO HotpotQA Natural Questions PAQ StackExchange Natural Language Inference SQuAD ArguAna BioASQ FiQA FEVER HoVer
Benchmarks: MTEB AIR-Bench 24.04
Metrics: nDCG@10 MAP V-measure accuracy Spearman correlation Recall@10 average MTEB score across 56 tasks
Baselines: E5-mistral-7b-instruct SFR-Embedding-2R SFR-Embedding-Mistral bge-en-icl Gte-Qwen2-7B-instruct Stella-1.5B-v5 Linq-Embed-Mistral text-embedding-3-large
Builds on: E5-mistral-7b-instruct Perceiver IO LoRA Mistral 7B
Closest papers by meaning
- Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.90)
- NUDGE: Lightweight Non-Parametric Fine-Tuning of Embeddings for Retrieval (ICLR 2025 · similarity 0.90)
- EmbedLLM: Learning Compact Representations of Large Language Models (ICLR 2025 · similarity 0.89)
- VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks (ICLR 2025 · similarity 0.89)
- A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation (CVPR 2025 · similarity 0.88)
- Making Text Embedders Few-Shot Learners (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.