Research map / Large language models / Language, safety and interpretability
Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Promptriever trains a LLaMA-2-7B bi-encoder retriever on MS MARCO augmented with ~500k LLM-generated instance-level instructions and instruction-negative passages, so that it can be prompted like a language model.
Problem
Standard dense retrievers are trained only for query-passage semantic similarity and lose the per-instance instruction following ability of their LM backbones, so they cannot have relevance criteria adjusted by natural language prompts.
Approach
Augment MS MARCO queries with diverse instance-level instructions that redefine relevance, plus generated "instruction negatives" (passages relevant to the query but not to the query+instruction), so the bi-encoder learns to condition relevance on the instruction; it also enables zero-shot prompt selection that improves retrieval.
Key results
- On FollowIR, Promptriever reaches p-MRR average +11.2 vs RepLLaMA -3.1 (+14.3) and score 26.1 vs 23.0, comparable to cross-encoder FollowIR-7B (+12.2 p-MRR).
- On InstructIR, Promptriever achieves Robustness@10 of 63.1 vs RepLLaMA 50.2 (+12.9).
- Selecting the best of 10 prompts raises Promptriever's BEIR average nDCG@10 from 55.0 to 56.4 (+1.4), while RepLLaMA and BM25 do not benefit.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with per-query instructions and instruction negatives rather than standard retrieval training | LLaMA 2 bi-encoder retriever · FollowIR average | p-MRR: -3.1 -> +11.2 | Helped |
| Train with per-query instructions and instruction negatives rather than standard retrieval training | LLaMA 2 bi-encoder retriever · InstructIR MS MARCO | Robustness@10: 50.2 -> 63.1 | Helped |
| Use the best of 10 natural-language prompts at test time | Promptriever · BEIR average | nDCG@10: 55.0 -> 56.4 | Helped |
| Use the best of 10 natural-language prompts at test time | RepLLaMA · BEIR average | nDCG@10: 54.9 -> 54.8 | Hurt |
| Use the best of 10 natural-language prompts at test time | BM25 · BEIR average | nDCG@10: 40.9 -> 35.9 | Hurt |
| Repeat the query to match the instruction length instead of adding meaningful per-query instructions (ablation) | Promptriever training recipe · FollowIR average | p-MRR: -3.1 -> -3.1 | No clear effect |
| Replace per-query instructions with generic retrieval instructions (ablation) | Promptriever training recipe · FollowIR average | p-MRR: -3.1 -> -3.0 | No clear effect |
| Randomly swap instructions across queries (ablation) | Promptriever training recipe · FollowIR average | p-MRR: -3.1 -> -0.9 | Helped |
| Add correctly paired per-query instructions (ablation) | Promptriever training recipe · FollowIR average | p-MRR: -0.9 -> +5.7 | Helped |
| Add instruction-negative passages to instruction training (ablation) | Promptriever training recipe · FollowIR average | p-MRR: +5.7 -> +8.8 | Helped |
| Train jointly on MS MARCO and instruction data (ablation) | Promptriever training recipe · FollowIR average | p-MRR: +8.8 -> +11.2 | Helped |
| Add instruction-negative passages to instruction training (ablation) | Promptriever training recipe · InstructIR MS MARCO | Robustness@10: 63.3 -> 62.0 | Hurt |
Surprising findings
- Prompts degrade RepLLaMA and BM25 performance, while Promptriever is the only model that reliably benefits from prompting.
- Generic instructions and swapped instructions give little or no instruction-following gain, showing that instruction length or lexical features alone do not explain the improvement.
Limitations
- Relies on LM-generated instructions and passages, which may contain errors, biases, or incorrect information.
- Only zero-shot prompting explored; in-context learning with example retrieval prompts not studied.
- Filtering by FollowIR-7B may bias the data; the reasons why some prompts help more than others remain unclear.
Experimental setup
Datasets: MS MARCO (tevatron-msmarco-aug) FollowIR InstructIR BEIR TREC DL19 TREC DL20
Benchmarks: FollowIR InstructIR BEIR MS MARCO TREC DL19 TREC DL20
Metrics: p-MRR MAP nDCG@5 nDCG@10 Robustness@10 MRR standard deviation across prompts
Baselines: RepLLaMA BM25 FollowIR-7B Mistral-7B-instruct MonoT5-3B TART-Contriever Instructor XL E5-Mistral Google Gecko
Builds on: RepLLaMA Tevatron FollowIR InstructIR TART
Closest papers by meaning
- Beyond Content Relevance: Evaluating Instruction Following in Retrieval Models (ICLR 2025 · similarity 0.88)
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation (ICML 2024 · similarity 0.87)
- System Prompt Optimization with Meta-Learning (NeurIPS 2025 · similarity 0.87)
- All You Need is One: Capsule Prompt Tuning with a Single Vector (NeurIPS 2025 · similarity 0.87)
- Investigating the Effectiveness of Task-Agnostic Prefix Prompt for Instruction Following (AAAI 2024 · similarity 0.87)
- The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.