Research map / Large language models / Retrieval and knowledge
Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LeReT, which samples diverse search queries via DSPy-bootstrapped few-shot prompts, then applies context distillation and IPO to train the query-generating LLM with retrieval-based rewards for multi-hop retrieval.
Problem
LLMs struggle to generate effective search queries for multi-hop retrieval used for grounding answers, and supervision for the best queries is costly to obtain.
Approach
Use diverse few-shot prompts (bootstrapped with DSPy) to generate exploratory search queries, score the retrieved documents with average precision as a per-hop reward, convert these into preference pairs, and optimize the query LLM with context distillation followed by IPO, treating the retriever and generator as black boxes.
Key results
- LeReT improves Llama 3 8b 1-hop recall on HotpotQA from 42.3 to 56.7 and 2-hop recall from 54.7 to 77.1 relative to the base model.
- On HoVer, LeReT raises Gemma 9b 4-hop recall from 50.1 to 79.4 and Llama 8b 4-hop recall from 50.0 to 76.9.
- With Llama 3.1 70b as generator, exact match rises from 38.1 (base) to 53.5 with LeReT retrieval; GPT-4 gains +17.3 EM points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train query generation with LeReT rather than few-shot prompting | Llama 3 8B Instruct · HotpotQA, 2 hops | recall: 64.8 -> 77.1 | Helped |
| Train query generation with LeReT rather than few-shot prompting | Llama 3 8B Instruct · HoVer, 4 hops | recall: 57.3 -> 76.9 | Helped |
| Train query generation with LeReT rather than few-shot prompting | Llama 3 8B Instruct; generator: Llama 3.1 70B · HotpotQA | exact match: 47.1 -> 52.5 | Helped |
| Apply a second LeReT iteration after the first | Gemma 2 9B Instruct · HotpotQA, 2 hops | recall: 78.2 -> 82.3 | Helped |
| Apply a second LeReT iteration after the first | Llama 3 8B Instruct · HoVer, 4 hops | recall: 73.5 -> 78.2 | Helped |
| Use diverse few-shot query sampling at temperature 0.7 instead of temperature 2.0 without few-shot prompting (ablation) | LeReT with Llama 3 8B Instruct · HotpotQA, 2 hops | recall: 69.86 -> 78.14 | Helped |
| Use diverse rather than fixed few-shot prompts at temperature 2.0 (ablation) | LeReT with Llama 3 8B Instruct · HotpotQA, 2 hops | recall: 70.79 -> 73.67 | Helped |
| Use diverse few-shot prompts at temperature 2.0 rather than diverse few-shot prompts at temperature 0.7 (ablation) | LeReT with Llama 3 8B Instruct · HotpotQA, 2 hops | recall: 78.14 -> 73.67 | Hurt |
| Use generator answer F1 rather than retrieval average precision as the reward (ablation) | LeReT with Llama 3 8B Instruct · HotpotQA, 2 hops | recall: 77.1 -> 56.8 | Hurt |
| Train on preferences from all hops rather than only the first hop (ablation) | LeReT with Llama 3 8B Instruct · HotpotQA, 2 hops | recall: 70.8 -> 77.1 | Helped |
| Train on preferences from all hops rather than only the first two hops (ablation) | LeReT with Llama 3 8B Instruct · HoVer, 4 hops | recall: 73.6 -> 76.9 | Helped |
| Use LeReT rather than few-shot prompting with Azure AI Search | Llama 3 8B Instruct · Hotpot, 2 hops | recall: 50.7 -> 60.0 | Helped |
Surprising findings
- Sampling at high temperature without few-shot prompting yields poor training data; few-shot prompting at temperature 0.7 gives the best RL data.
- Using generator F1 as the reward produces many wrong preference pairs (over 50%) and underperforms the retrieval-based reward.
Limitations
- Relies on direct supervision (gold supporting documents) for the retrieval reward; indirect generator-based reward performs worse.
- Only the query-generating LLM is trained; retriever and generator are treated as black boxes and not updated.
- Evaluated mainly on two Wikipedia-based multi-hop datasets; long-form results are preliminary.
Experimental setup
Datasets: HotpotQA HoVer Wikipedia 2017 abstracts (ColBERTv2 index)
Benchmarks: HotpotQA HoVer
Metrics: recall average precision exact match F1
Baselines: Base model (Llama 3 8b, Gemma 2 9b) Few-shot prompting Few-shot all LeReT-CD High-temperature sampling
Builds on: DSPy IPO DPO iterative DPO ColBERTv2
Closest papers by meaning
- ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Efficient Exploration for LLMs (ICML 2024 · similarity 0.88)
- Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay (NeurIPS 2025 · similarity 0.88)
- EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration (ICML 2025 · similarity 0.88)
- Can large language models explore in-context? (NeurIPS 2024 · similarity 0.87)
- LLM Alignment as Retriever Optimization: An Information Retrieval Perspective (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.