Research map / Large language models / Retrieval and knowledge
Trustworthy Alignment of Retrieval-Augmented Large Language Models via Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Trustworthy-Alignment, a PPO-based RL method that rewards LLMs for answering from counterfactual retrieved context rather than parametric memory, without supervision on how to respond.
Problem
Retrieval-augmented LLMs hallucinate when contextual evidence conflicts with parametric knowledge; the goal is to make them rely on the retrieved context.
Approach
Treat RAG generation as an MDP and reward the model with a sentence-level signal for producing the substituted (context) answer and penalize producing the original (parametric) answer, aligning via PPO with KL, collapse penalties and annealed temperature exploration; theoretically argued via potential-based reward shaping that the trust reward's optimal policy is preserved.
Key results
- On Natural Questions, Llama-2-7b-chat with Trustworthy-Alignment reaches 94.92% EM and 0.70% memorization ratio, versus 62.17% EM / 24.54% M_R for the best prompting baseline (GPT-4 opinion prompt reports 89.71% EM).
- Trustworthy-Alignment improves Llama-2-7b accuracy on ConflictQA-PopQA over prompting only where SFT gets 76.19%; Llama-2-7b-chat SFT reaches 85.31% vs 66.14% for the RL method on PopQA.
- On Open LLM Leaderboard, Vicuna-7b-v1.5 drops by under 1% average after alignment, and Llama-2-7b-chat gains on ARC (+3.50) and MMLU (+2.99).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply Trustworthy-Alignment instead of SFT | Llama-2-7b-chat · Natural Questions | EM (%): 92.42 -> 94.92 | Helped |
| apply Trustworthy-Alignment instead of SFT | Llama-2-7b-chat · ConflictQA-PopQA | Acc (%): 85.31 -> 66.14 | Hurt |
| apply Trustworthy-Alignment instead of SFT | Llama-2-7b-chat · ConflictQA-StrategyQA | Acc (%): 73.70 -> 64.87 | Hurt |
| apply Trustworthy-Alignment instead of SFT | Llama-2-7b · ConflictQA-PopQA | Acc (%): 76.19 -> 81.19 | Helped |
| apply Trustworthy-Alignment instead of SFT | Llama-2-7b · ConflictQA-StrategyQA | Acc (%): 67.38 -> 69.17 | Helped |
| disable annealing exploration and use fixed temperature 1.0 (ablation) | Trustworthy-Alignment on Llama-2-7b-chat · Natural Questions | EM (%): 94.92 -> 96.12 | Helped |
| disable annealing exploration and use fixed temperature 1.0 (ablation) | Trustworthy-Alignment on Llama-2-7b-chat · Natural Questions | M_R (%): 0.70 -> 1.10 | Hurt |
| use a 1.3B actor instead of a 7B actor (ablation) | Trustworthy-Alignment · Natural Questions | EM (%): 94.92 -> 24.04 | Hurt |
| use a 13B actor instead of a 7B actor (ablation) | Trustworthy-Alignment · Natural Questions | EM (%): 94.92 -> 94.39 | No clear effect |
| replace the pretrained critic with a randomly initialized critic (ablation) | Trustworthy-Alignment · Natural Questions | EM (%): 94.92 -> 93.58 | No clear effect |
| replace the 350M critic with a larger Llama-2-7b-chat critic (ablation) | Trustworthy-Alignment · Natural Questions | EM (%): 94.92 -> 93.32 | No clear effect |
| modify the trust reward to also reward the original answer (ablation) | Trustworthy-Alignment on Llama-2-7b-chat · parametric-knowledge alignment | p^o (%): 0.67 -> 51.74 | Helped |
Surprising findings
- Trustworthy-Alignment preserves the original model's language style, whereas SFT produces terse answers; human raters judged RL outputs more fluent.
- Replacing the critic with a random or larger critic had little effect on performance.
Limitations
- Performance on Llama-2-7b and Vicuna-7b-v1.5 is below SFT on Natural Questions due to weaker instruction-following of base models.
- Assumes the retriever returns contexts containing correct answers; a modified reward that also rewards parametric answers is a partial remedy.
- Both SFT and RL methods degrade noticeably on GSM8k.
Experimental setup
Datasets: Natural Questions ConflictQA-PopQA ConflictQA-StrategyQA Open LLM Leaderboard tasks
Benchmarks: Natural Questions ConflictQA-PopQA ConflictQA-StrategyQA AI2 Reasoning Challenge HellaSwag MMLU TruthfulQA Winogrande GSM8k
Metrics: exact match memorization ratio accuracy win rate
Baselines: ChatGPT GPT-4 Instruction prompting Opinion prompting SFT
Builds on: PPO RLHF potential-based reward shaping DeepSpeed-Chat
Closest papers by meaning
- LLM Alignment as Retriever Optimization: An Information Retrieval Perspective (ICML 2025 · similarity 0.90)
- Doubly Robust Alignment for Large Language Models (NeurIPS 2025 · similarity 0.89)
- More RLHF, More Trust? On The Impact of Preference Alignment On Trustworthiness (ICLR 2025 · similarity 0.89)
- Advantage-Guided Distillation for Preference Alignment in Small Language Models (ICLR 2025 · similarity 0.89)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.89)
- Understanding the Learning Dynamics of Alignment with Human Feedback (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.