Research map / Large language models / Instruction and fine-tuning
Beyond Content Relevance: Evaluating Instruction Following in Retrieval Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces InfoSearch, a benchmark for evaluating whether retrieval models follow document-level instructions (audience, keyword, format, language, length, source) beyond content relevance, with two new metrics (SICR and WISE), and evaluates 15 retrieval and reranking models.
Problem
Instruction-aware retrievers mainly target content relevance and task-level instructions, neglecting user preferences over document-level attributes such as audience, format, length, language, and source; existing IR metrics like nDCG and p-MRR do not faithfully measure instruction following.
Approach
A benchmark with six document-level instruction dimensions and instructed and reversely instructed modes, paired with rank-based metrics (SICR as a strict compliance ratio and WISE as a weighted reward/penalty score) that track gold-document rank changes across the original, instructed, and reversed modes.
Key results
- GPT-4o as list-wise reranker achieves the highest WISE (33.5) and SICR (32.1) among all 15 models.
- Dense retrievers mostly score negative WISE; GritLM is the best dense model with WISE -11.1 and SICR 6.9.
- Reranking models generally outperform dense retrieval models, and larger models typically score higher on WISE and SICR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use list-wise reranking instead of dense retrieval | Mistral-Instruct-v0.2 · InfoSearch (six-dimension average) | WISE: -49.2 -> 8.1 | Helped |
| Use list-wise reranking instead of dense retrieval | Mistral-Instruct-v0.2 · InfoSearch (six-dimension average) | SICR: 0.0 -> 22.0 | Helped |
| Use point-wise reranking instead of dense retrieval | Mistral-Instruct-v0.2 · InfoSearch (six-dimension average) | WISE: -49.2 -> 4.1 | Helped |
| Use point-wise reranking instead of dense retrieval | Mistral-Instruct-v0.2 · InfoSearch (six-dimension average) | SICR: 0.0 -> 8.1 | Helped |
| Use list-wise rather than point-wise reranking | Mistral-Instruct-v0.2 · InfoSearch (six-dimension average) | WISE: 4.1 -> 8.1 | Helped |
| Use list-wise rather than point-wise reranking | Mistral-Instruct-v0.2 · InfoSearch (six-dimension average) | SICR: 8.1 -> 22.0 | Helped |
| Use list-wise rather than point-wise reranking | retrieval models · InfoSearch | WISE | Helped |
| Use list-wise rather than point-wise reranking | retrieval models · InfoSearch | SICR | Helped |
| Use reranking instead of dense retrieval | retrieval models · InfoSearch | WISE | Helped |
| Use reranking instead of dense retrieval | retrieval models · InfoSearch | SICR | Helped |
| Use RankZephyr rather than Zephyr-beta | list-wise reranking · InfoSearch (six-dimension average) | WISE: -3.2 -> 14.5 | Helped |
| Use RankZephyr rather than Zephyr-beta | list-wise reranking · InfoSearch (six-dimension average) | SICR: 8.7 -> 10.5 | Helped |
Surprising findings
- p-MRR ranks bge-large-v1.5 and Instructor above GPT-4o, contradicting the actual gold-document rank trends, while WISE tracks them.
- Most models, including fine-tuned ones, score lower than BM25 under p-MRR, and format and audience are consistently hard dimensions.
Limitations
- Dataset is small (600 core queries, 6,392 documents) and some documents were rewritten by GPT-4.
- Mistral zero-shot and GPT-4o are evaluated with prompting, and the paper provides no compute or cost details.
- Paper notes that most models still fall short of instruction compliance, leaving the proposed benchmark only partially solved.
Experimental setup
Datasets: BioASQ SciFact MS MARCO PublicHealthQA medical_qa CNN English news StackOverflow
Benchmarks: InfoSearch
Metrics: nDCG@10 Robustness@10 p-MRR WISE SICR
Baselines: BM25 BGE-Large-v1.5 E5-Large-v2 Instructor-XL E5-Mistral GritLM NV-Embed-v2 GTE-Qwen2 SFR-Embedding-v2 FollowIR RankVicuna RankZephyr
Builds on: FollowIR p-MRR RankGPT InstructIR
Closest papers by meaning
- Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.88)
- Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models (ICLR 2025 · similarity 0.88)
- DINGO: Towards Diverse and Fine-Grained Instruction-Following Evaluation (AAAI 2024 · similarity 0.88)
- Toward Verifiable Instruction-Following Alignment for Retrieval Augmented Generation (AAAI 2025 · similarity 0.87)
- Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation (ICML 2024 · similarity 0.86)
- Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.