Research map / Large language models / Retrieval and knowledge
InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Scales retrieval-augmented pretraining (Retro) to a 48B GPT-based model via continued pretraining on 100B tokens over a 1.2T-token retrieval database, then instruction-tunes it (InstructRetro) and shows zero-shot gains over an instruction-tuned GPT with RAG.
Problem
Pretrained retrieval-augmented LLMs have been small (around 10B parameters, e.g., Retro 7.5B), which limits the effectiveness of instruction tuning and zero-shot generalization.
Approach
Continue pretraining a 43B GPT with Retro-style retrieval (unfreezing all decoder parameters, retrieving from 1.2T tokens), then instruction-tune with the retrieval encoder gated off, so the resulting decoder can be used alone as a GPT-style model.
Key results
- InstructRetro 43B improves on instruction-tuned GPT-43B (with RAG) by about 7% average relative across 8 short-form QA tasks, 10% across 4 long-form QA tasks, and 16% across 3 summarization tasks.
- Retro 48B reaches lower perplexity than GPT 43B trained on 1.2T tokens while adding only 2.58% additional GPU hours.
- The decoder-only InstructRetro 43B (encoder ablated) achieves results very comparable to InstructRetro 48B, and on several short-form QA tasks outperforms RA-DIT 65B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Continue GPT pretraining with retrieval, then instruction-tune | InstructRetro 43B vs InstructGPT 43B · Short-form QA and reading comprehension (8 tasks) | relative improvement: 7% | Helped |
| Continue GPT pretraining with retrieval, then instruction-tune | InstructRetro 43B vs InstructGPT 43B · Long-form QA (4 tasks) | relative improvement: 10% | Helped |
| Continue GPT pretraining with retrieval, then instruction-tune | InstructRetro 43B vs InstructGPT 43B · Summarization (3 tasks) | ROUGE score improvement: 16% | Helped |
| Remove the Retro encoder during evaluation (ablation) | InstructRetro · Short-form QA and reading comprehension | task scores | No clear effect |
| Remove the Retro encoder during evaluation (ablation) | InstructRetro · Long-form QA | F1 | No clear effect |
| Unfreeze the Retro decoder during retrieval-augmented continued pretraining (ablation) | Retro-fitting · Held-out validation set | perplexity | Helped |
| Increase the number of Retro encoder layers (ablation) | Retro · Held-out validation set | perplexity | No clear effect |
| Apply instruction tuning to retrieval-augmented pretrained Retro (ablation) | Retro · Natural Questions | EM | Helped |
| Apply instruction tuning to pretrained GPT (ablation) | GPT · Natural Questions | EM | Helped |
| Apply product quantization instead of flat encoding (ablation) | Faiss retrieval index · Full pretraining corpus and Wikipedia corpus | query speed | Helped |
| Increase the nearest-neighbor count K (ablation) | Faiss retrieval index · Full pretraining corpus and Wikipedia corpus | query speed | No clear effect |
| Use efSearch=16, nprobe=4096 instead of efSearch=32, nprobe=4096 (ablation) | Faiss retrieval index · Pretraining corpus retrieval | recall accuracy at K=2000: 0.93 to 0.88 | Hurt |
Surprising findings
- Ablating the retrieval encoder after instruction tuning leaves a decoder-only model that performs comparably to the full model.
- Unfreezing the decoder during retro-fitting converges faster and yields better validation perplexity than freezing it, contrary to prior Retro-fitting practice.
Limitations
- Retro encoder is disabled during instruction tuning and evaluation, so the retrieval-augmented encoder path is not exploited after tuning.
- Instruction tuning data lacks retrieval neighbors, so retrieval-augmented instruction data was not used.
- Weaker performance on coding and role-play in MT-Bench, attributed to pretraining corpus coverage.
Experimental setup
Datasets: Pretraining corpus of 1.2T English tokens SODA ELI5 Self-Instruct Unnatural Instructions FLAN Chain-of-thought datasets OpenAssistant Dolly NQ TriviaQA NewsQA
Benchmarks: NQ TriviaQA NewsQA SQuAD 2.0 SQuAD 1.1 Quoref NarrativeQA DROP doc2dial QMSum SummScreenFD GovReport
Metrics: exact match F1 ROUGE perplexity MT-Bench score
Baselines: GPT-3 175B PaLM 2-L GLaM 64B FLAN-LaMDA 137B Llama 2 RAG 70B Retro 7.5B Retro++ 9B Atlas 11B Raven 11B RA-DIT 65B InstructGPT-43B (GPT-fitting + SFT + RAG)
Builds on: Retro Retro-fitting Faiss DRAGON+
Closest papers by meaning
- RA-DIT: Retrieval-Augmented Dual Instruction Tuning (ICLR 2024 · similarity 0.89)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.89)
- InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct (AAAI 2025 · similarity 0.88)
- Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructions (ICLR 2024 · similarity 0.88)
- PRESTO: Preimage-Informed Instruction Optimization for Prompting Black-Box LLMs (NeurIPS 2025 · similarity 0.88)
- RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.