Research map / Large language models / Alignment and preferences
Interpreting Language Reward Models via Contrastive Explanations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes contrastive (counterfactual and semifactual) explanations for binary preference comparisons made by language reward models, generating LLM-perturbed responses along high-level evaluation attributes to explain local and global RM behaviour.
Problem
Reward models used in LLM alignment are black boxes whose scalar rewards lack direct meaning, so their pairwise preference predictions are hard to explain and trust.
Approach
Adapts counterfactual and semifactual explanations from classification to RM binary comparisons: an LLM, prompted with a two-step attribute-conditioned strategy, perturbs chosen/rejected responses in the opposing direction along 15 high-level attributes; perturbations that flip the RM preference are CFs, others SFs. Aggregating these gives global attribute sensitivity and representative examples.
Key results
- On harmless, OURS finds CFs for both responses in 69% of test comparisons versus at most 26% for baselines (PJ, RP).
- Perturbations from OURS have much lower semantic distance than baselines across all datasets (e.g. 0.07 vs 0.18 for RP on hs2).
- v2 is most sensitive to harmlessness-related attributes on the harmless dataset, consistent with it being trained on that data; 8B RMs also prefer disrespectful perturbations in some cases (win rate 0.15-0.30 for LlamaOB and LlamaSW).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| attribute-conditioned two-step LLM perturbation (OURS) vs. Polyjuice and random LLM perturbation | contrastive explanations for RMs · harmless, helpful, hs2 | CF coverage (both responses): OURS 0.69/0.80/0.33 vs best baseline 0.26/0.21/0.04 | Helped |
| attribute-conditioned perturbation (OURS) vs. baselines | contrastive explanations for RMs · harmless, helpful, hs2 | semantic distance to original (lower better): harmless 0.36 vs 0.69 (PJ); hs2 0.07 vs 0.18 (RP) | Helped |
| attribute-conditioned perturbation (OURS) vs. baselines | contrastive explanations for RMs · harmless, helpful, hs2 | semantic diversity: OURS 0.31/0.20/0.07 vs baselines up to 0.49/0.46/0.14 | Hurt |
| SF coverage with attribute-conditioned perturbation (OURS) vs. baselines | contrastive explanations for RMs · harmless, helpful, hs2 | SF coverage (both): OURS 0.90 vs PJ 0.89 on harmless; 0.74 vs 0.87 PJ on helpful | Mixed |
| center-prompt (perturbations centred on Step-1 relevant words) vs. pass-prompt (no constraint) (ablation) | perturbation prompt · harmless | CF coverage (both) and semantic distance: CF cov 0.68 vs 0.64; sem dist 0.36 vs 0.48 | Helped |
| center-prompt vs only-prompt (strict word-restricted editing) (ablation) | perturbation prompt · harmless | CF coverage (both): 0.68 vs 0.49 | Helped |
| center-prompt vs only-prompt (ablation) | perturbation prompt · harmless | semantic diversity: 0.31 vs 0.21 | Helped |
| Step-1 word-focusing constraint in the prompt (ablation) | perturbation prompt · hs2 | CF coverage (both): 0.33 (center) vs 0.28 (only), 0.39 (pass) | Helped |
| attribute-conditioned perturbation sensitivity: v2 more sensitive to harmful-content attributes on harmless dataset | DeBERTa-v2 RM · harmless, Y+ perturbations | preference flip rate: v1-v2 ranking similarity 0.16 vs 0.60-0.71 on other datasets | Mixed |
| adding a disrespectful phrase (the hell) via appropriateness perturbation to responses | pythia-1.4b RM · harmless, 110 comparisons | disrespectful perturbation win rate (lower better): 0.75 | Hurt |
| larger 8B RMs (LlamaOB, LlamaSW) vs smaller DeBERTa RMs | RM scale · harmless, 110 comparisons | disrespectful perturbation win rate (lower better): LlamaSW 0.15 and LlamaOB 0.30 vs v1 0.51, v2 0.56 | Helped |
Surprising findings
- Larger 8B RMs (LlamaOB, LlamaSW) still sometimes prefer responses with inserted disrespectful phrasing, and LlamaSW assigned higher reward to an answer asserting it can help with harmful activities.
Limitations
- No guarantee that counterfactual responses exist for every comparison, since perturbations are generated before being tested on the RM.
- Perturbing mechanism is not linked to the model being explained; SFs do not distinguish between reward levels between the two originals and beyond.
- Results depend on the choice of LLM generator (GPT-4o) and the attribute list, and perturbations are less semantically diverse than baselines.
Experimental setup
Datasets: HelpSteer2 HH-RLHF helpful HH-RLHF harmless
Benchmarks: HelpSteer2 HH-RLHF helpful HH-RLHF harmless
Metrics: CF coverage SF coverage syntactic distance semantic distance semantic diversity preference flip rate Kendall's tau disrespectful perturbation win rate
Baselines: Polyjuice random LLM-based perturbation
Builds on: Polyjuice Bradley-Terry reward modelling Sentence-BERT
Closest papers by meaning
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.92)
- Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree? (ICLR 2025 · similarity 0.90)
- Transforming and Combining Rewards for Aligning Large Language Models (ICML 2024 · similarity 0.90)
- RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.90)
- Post-hoc Reward Calibration: A Case Study on Length Bias (ICLR 2025 · similarity 0.90)
- RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.