Research map / Large language models / Alignment and preferences

Explainable Reinforcement Learning from Human Feedback to Improve Alignment

NeurIPS 2025 · Large language models › Alignment and preferences · approach: Reward modeling for diverse preferences · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes XRLHF, a post-hoc method that explains unsatisfactory RLHF responses by retrieving a convex-combination decomposition of training preference pairs, then unlearns those pairs in the reward model and fine-tunes the policy with a KL constraint on satisfactory prompts.

Problem

RLHF-tuned language models still produce unsatisfactory responses (harmful, inaccurate, redundant), often because the preference data contains misleading or noisy labels; there is no principled way to trace and correct the training data responsible.

Approach

Explain an unsatisfactory prompt-response pair by finding a minimal-distance subset of preference-comparison feature vectors whose convex hull contains the projected pair feature, select them with an iterative nearest-neighbor algorithm, then unlearn them via negative gradient on the reward model and fine-tune the policy under the unlearned reward with a KL constraint on satisfactory prompts.

Key results

• PPO+XRLHF achieves 76.4% win rate over SFT on full-hh-rlhf with opt-1.3B (reward model) versus 68.3% for PPO alone.
• ReMax+XRLHF reaches 80.1% GPT-4 win rate over SFT on full-hh-rlhf with pythia-2.8B, versus 66.8% for ReMax.
• On TL;DR with Llama-2-7B, ReMax+XRLHF gets 77.4% reward-model win rate over SFT versus 69.6% for ReMax.

Measured effects reported in the paper

Surprising findings

• Only about 2-3% of reward-training data is identified as explaining unsatisfactory responses, so unlearning is far cheaper than retraining.
• Gains on the full validation set are smaller than on the unsatisfactory subset, indicating slight degradation on satisfactory prompts.

Limitations

• Explanation is derived for linear reward models; general case relies on a heuristic using final-layer features.
• Unsatisfactory-response labels depend on a reward-model threshold calibrated by human evaluation on only 50 samples.
• Slight degradation on satisfactory prompts in the full validation set.

Experimental setup

Datasets: full-hh-rlhf TL;DR summarization

Benchmarks: full-hh-rlhf test set TL;DR test set

Metrics: win rate over SFT model win rate over base RLHF (PPO, ReMax) on unsatisfactory and f

Baselines: PPO ReMax

Builds on: RLHF PPO ReMax DPO negative gradient unlearning

Closest papers by meaning

• Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models (ICLR 2024 · similarity 0.90)
• MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.90)
• Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.90)
• SELF-EVOLVED REWARD LEARNING FOR LLMS (ICLR 2025 · similarity 0.90)
• RLTHF: Targeted Human Feedback for LLM Alignment (ICML 2025 · similarity 0.90)
• SEAL: Systematic Error Analysis for Value ALignment (AAAI 2025 · similarity 0.90)