Research map / Large language models / Alignment and preferences
RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RATE, a double-rewrite estimator that uses LLM rewrites of rewrites to cancel off-target rewrite errors and estimate causal (ATE/ATT/ATU) effects of text attributes on reward models.
Problem
Reward models are black boxes, and naive comparisons of rewards across attribute-labeled data conflate the target attribute with spurious correlated attributes; imperfect LLM counterfactual rewrites also introduce off-target changes that bias causal estimates.
Approach
Estimate the causal effect of a binary attribute on a reward by comparing rewrites to rewrites-of-rewrites, so that off-target changes introduced by the rewriter cancel in expectation under assumptions that rewrite errors do not depend on rewrite direction and reward is additive in rewrite errors.
Key results
- On semi-synthetic typo experiments with known zero effect, the double-rewrite RATE estimate stays near zero as typo correlation grows while the naive and single-rewrite estimators become increasingly negatively biased.
- On the DistilBERT sentiment-classifier length experiment, the single-rewrite estimator drifts increasingly negative with length-sentiment correlation while RATE stays near zero.
- On real reward models, naive estimates differ substantially from RATE; length bias appears largely an artifact of naive evaluation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use double rewrites instead of the naive estimator | RATE · IMDB, starts with a vowel | causal effect estimate accuracy | Helped |
| Use double rewrites instead of single rewrites (ablation) | RATE · IMDB, starts with a vowel | causal effect estimate accuracy | Helped |
| Use the naive estimator rather than RATE | Reward-model attribute evaluation · IMDB, starts with a vowel | invariance to spurious correlation | Hurt |
| Use single rewrites instead of double rewrites (ablation) | Rewrite-based estimator · IMDB, starts with a vowel | invariance to spurious correlation | Hurt |
| Use RATE instead of the naive estimator | DistilBERT sentiment classifier · IMDB, length | invariance to distributional shift | Helped |
| Use single rewrites instead of double rewrites (ablation) | Rewrite-based estimator · IMDB, length | invariance to distributional shift | Hurt |
| Use RATE instead of the naive estimator | Reward models on HelpSteer · HelpSteer, complexity and helpfulness | invariance to distributional shift | Helped |
| Use double rewrites instead of single rewrites (ablation) | Rewrite-based estimator · HelpSteer, helpfulness | stability under distributional shift | Helped |
| Use RATE instead of the naive estimator | ArmoRM · HelpSteer, helpfulness | stability of treatment effect | Helped |
| Use single rewrites instead of double rewrites (ablation) | Rewrite-based estimator · HelpSteer, helpfulness | stability of treatment effect | Hurt |
| Use RATE instead of the naive estimator | Reward models across IMDB, ELI5, and HelpSteer · Reward-model attributes | estimated attribute effects | Mixed |
| Use double rewrites instead of single rewrites (ablation) | RATE · HelpSteer complexity; HelpSteer helpfulness; ELI5 and IMDB length; IMDB and HH-RLHF sentiment | estimated treatment effects | Mixed |
Surprising findings
- GPT-4o corrects typos even when not asked to, introducing bias into single-rewrite estimators.
- The apparent length bias in reward models may be mainly an artifact of confounded naive evaluation.
Limitations
- Validity depends on rewrite quality, which has no ground truth and is judged qualitatively.
- Estimates measure reward model sensitivity in isolation, not the downstream effect on aligned LLM behavior.
- Requires assumptions (rewrite errors independent of direction; additive reward decomposition) that are sufficient rather than verified.
Experimental setup
Datasets: IMDB ELI5 HelpSteer HH-RLHF
Benchmarks: IMDB ELI5 HelpSteer HH-RLHF RewardBench
Metrics: average treatment effect average treatment effect on the treated average treatment effect on the untreated Cohen's d 95% confidence intervals
Baselines: naive conditional reward difference single-rewrite estimator RewardBench
Builds on: average treatment effect estimation CausaLM Polyjuice
Closest papers by meaning
- Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree? (ICLR 2025 · similarity 0.86)
- Preference Learning for AI Alignment: a Causal Perspective (ICML 2025 · similarity 0.84)
- RRM: Robust Reward Model Training Mitigates Reward Hacking (ICLR 2025 · similarity 0.84)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.84)
- Diverging Preferences: When do Annotators Disagree and do Models Know? (ICML 2025 · similarity 0.84)
- Towards Characterizing Domain Counterfactuals for Invertible Latent Causal Models (ICLR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.