Research map / Large language models / Instruction and fine-tuning
Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies compute-optimal contrastive fine-tuning of Pythia decoder-only LMs into text embedders across full FT, block freezing, bias-only tuning and LoRA, fits scaling laws, and derives a budget-based recipe choosing full FT below ~9e16 FLOP and LoRA above.
Problem
Given a fixed training compute budget, which decoder-only LLM size, data quantity, and parameter-efficient fine-tuning method yield the best contrastively trained text embedding model; this has not been studied systematically.
Approach
Runs IsoFLOP grid searches over model size, data and fine-tuning method, fits an extended scaling law that includes the fraction of trainable parameters, and turns the resulting compute-optimal frontier into an algorithm that selects full fine-tuning or LoRA (with rank) for a given budget.
Key results
- Full fine-tuning is optimal below about 9.06e16 FLOP and LoRA above, per fitted frontier lines (ln loss = -0.21 ln budget + 8.39 for full FT; -0.22 ln budget + 8.93 for LoRA).
- LoRA ranks 32 or 128 are optimal in almost all settings; bias-only tuning is consistently worse, with loss above 0.4.
- Spearman correlation between training loss and MTEB subset performance is -0.892.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use mean pooling instead of last-token pooling (ablation) | Pythia embedding models · MTEB (without MSMarco subset) | Avg. score: 0.36 -> 0.47 | Helped |
| Set Temperature^-1 to 40 rather than 50 (ablation) | Partial-tuning embedding models · MTEB subset | Avg. score: 0.5 -> 0.51 | Helped |
| Set Temperature^-1 to 40 rather than 35 or 30 (ablation) | Partial-tuning embedding models · MTEB subset | Avg. score: 0.5 -> 0.51 | Helped |
| Set Temperature^-1 to 40 rather than 10 (ablation) | Partial-tuning embedding models · MTEB subset | Avg. score: 0.47 -> 0.51 | Helped |
| Set Temperature^-1 to 40 rather than 1 (ablation) | Partial-tuning embedding models · MTEB subset | Avg. score: 0.3 -> 0.51 | Helped |
| Use learning rate 1e3 instead of 1e2 (ablation) | LoRA rank 8 · LoRA learning-rate grid | Loss: 1.96 -> 1.35 | Helped |
| Use learning rate 1e3 instead of 1e4 (ablation) | LoRA rank 8 · LoRA learning-rate grid | Loss: 2.22 -> 1.35 | Helped |
| Use learning rate 1e3 instead of 1e2 (ablation) | LoRA rank 16 · LoRA learning-rate grid | Loss: 1.96 -> 1.34 | Helped |
| Use learning rate 1e3 instead of 1e4 (ablation) | LoRA rank 16 · LoRA learning-rate grid | Loss: 2.22 -> 1.34 | Helped |
| Use learning rate 1e3 instead of 1e2 (ablation) | LoRA rank 32 · LoRA learning-rate grid | Loss: 2.09 -> 1.32 | Helped |
| Use learning rate 1e3 instead of 1e4 (ablation) | LoRA rank 32 · LoRA learning-rate grid | Loss: 2.19 -> 1.32 | Helped |
| Use learning rate 1e2 instead of 1e3 (ablation) | Bias tuning · Bias-tuning learning-rate grid | Loss: 1.37 -> 1.24 | Helped |
Surprising findings
- Bias-only tuning is consistently worse than other methods under IsoFLOP comparison.
- Optimal configurations often do not use the largest model that fits the budget.
Limitations
- Only Pythia models (all trained on 300B tokens) were used for fitting; generalisation checked with a single Gemma 2B and Gemma 2 2B model.
- Each experiment was run once without random seed averaging.
- Only mean pooling was studied and inference cost was not considered.
Experimental setup
Datasets: BAAI BGE English dataset The Pile
Benchmarks: MTEB subset (EmotionClassification, TwentyNewsgroupsClustering, SprintDuplicateQuestions, AskUbuntuDupQuestions, SciFact
Metrics: contrastive training loss MTEB subset score
Baselines: full fine-tuning bias-only tuning LoRA rank 16 (prior work) block freezing
Builds on: Chinchilla scaling laws LoRA Pythia GradCache BGE
Closest papers by meaning
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR 2025 · similarity 0.90)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.90)
- MMTEB: Massive Multilingual Text Embedding Benchmark (ICLR 2025 · similarity 0.90)
- EmbedLLM: Learning Compact Representations of Large Language Models (ICLR 2025 · similarity 0.89)
- Analyzing Similarity Metrics for Data Selection for Language Model Pretraining (NeurIPS 2025 · similarity 0.89)
- The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.