Research map / Large language models / Retrieval and knowledge
Generative Representational Instruction Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GRIT, which finetunes a single LLM (GritLM 7B and 8x7B) on both embedding data (contrastive loss) and generative data (next-token loss), distinguished by instructions, unifying text embedding and generation without performance loss.
Problem
Existing LLMs perform well at either generation or text embedding but not both, so separate models are needed for tasks like RAG, reranking, and serving.
Approach
Jointly finetune one pretrained LLM with a contrastive representational loss (bidirectional attention, mean pooling, in-batch negatives from the same dataset) on embedding data and a language-modeling loss on generative data, with instructions/format tokens telling the model which objective applies; also mixes sample- and token-level generative loss aggregation.
Key results
- GritLM 7B reaches 66.8 MTEB average, state of the art among open models at the time (56 datasets).
- GritLM 8x7B has a generative average of 65.7, the best among open generative models tried, outperforming Llama 2 70B-based models.
- Query Caching and Doc Caching give >60% faster RAG for long (4000-token) documents on CPU, with Doc Caching matching or slightly exceeding RAG match score (33.38 vs 30.47).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| unify embedding and generation instruction tuning | Mistral 7B · MTEB (56 datasets) | average: 66.8 vs 66.8 | No clear effect |
| unify embedding and generation instruction tuning | Mistral 7B · Generative task average | average: 55.2 -> 55.5 | Helped |
| scale unified model to 8x7B | GritLM · Generative task average | average: 55.5 -> 65.7 | Helped |
| scale unified model from 7B to 8x7B | GritLM · MTEB (56 datasets) | average: 66.8 -> 65.7 | Hurt |
| apply KTO alignment after GRIT (ablation) | GritLM 7B · MTEB (56 datasets) | average: 66.8 to 66.7 | Hurt |
| apply KTO alignment after GRIT (ablation) | GritLM 7B · Generative task average | average: 55.5 -> 57.4 | Helped |
| use bidirectional attention and mean pooling for embedding (ablation) | GritLM embedding pathway · MTEB (56 datasets) | average: 60.0 -> 61.8 | Helped |
| replace causal attention with bidirectional attention for generative instructions (ablation) | Generative-only model · Generative task average | average: 55.2 -> 50.7 | Hurt |
| replace Mistral 7B initialization with Llama 2 7B (ablation) | GRIT · MTEB (56 datasets) | average: 54.6 -> 48.2 | Hurt |
| replace Mistral 7B initialization with GPT-J 6B (ablation) | GRIT · Generative task average | average: 22.4 -> 14.0 | Hurt |
| use E5 instead of MEDI2 for embedding training (ablation) | GritLM · MTEB (56 datasets) | average: 64.7 -> 66.0 | Helped |
| replace Tülü 2 with OASST for generative training (ablation) | GritLM · Generative task average | average: 55.2 -> 37.7 | Hurt |
Surprising findings
- Few-shot examples in the embedding format worsen embedding performance overall.
- Embedding performance of a pretrained model before finetuning is not predictive of embedding performance after GRIT finetuning; generative performance is a better indicator.
Limitations
- Requires more finetuning compute than generative-only instruction tuning due to two objectives.
- GritLM 7B is more costly to run and produces 4096-dim embeddings versus smaller embedding models like BGE Large.
- Few-shot embedding does not work; Query and Doc-Query caching lose performance due to attention mismatch.
Experimental setup
Datasets: E5 S2ORC Tulu 2 MEDI MEDI2 UltraFeedback OASST UltraChat Natural Questions BEIR NQ corpus
Benchmarks: MTEB MMLU GSM8K BBH TyDi QA HumanEvalSynthesize AlpacaEval 1.0 Natural Questions RAG
Metrics: nDCG@10 V-measure accuracy average precision MAP Spearman exact match F1 pass@1 % win
Baselines: OpenAI v3 E5 Mistral 7B BGE Large Instructor XL Llama 2 70B Mixtral 8x7B Instruct Tulu 2 70B Zephyr 7B beta SGPT BE 5.8B
Builds on: SGPT E5 Tulu 2 MEDI Mistral 7B
Closest papers by meaning
- GRACE: Generative Representation Learning via Contrastive Policy Optimization (ICLR 2026 · similarity 0.87)
- Instruction Tuning With Loss Over Instructions (NeurIPS 2024 · similarity 0.86)
- UME-R1: Exploring Reasoning-Driven Generative Multimodal Embeddings (ICLR 2026 · similarity 0.86)
- Learning to Rank in Generative Retrieval (AAAI 2024 · similarity 0.86)
- RA-DIT: Retrieval-Augmented Dual Instruction Tuning (ICLR 2024 · similarity 0.86)
- Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.