Research map / Large language models / Reasoning and chain of thought

Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correction

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Process reward models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Causal Reward Adjustment (CRA), which trains sparse autoencoders on PRM activations, identifies reward-hacking-associated latent features via t-tests, and applies backdoor adjustment to correct PRM scores during beam search for math reasoning.

Problem

Process reward models used in external reasoning (beam search with a PRM) often give high scores to logically incorrect reasoning steps, so the selected paths are wrong; this is reward hacking, which the authors attribute to confounding semantic features.

Approach

Model reward hacking as a backdoor-path confounding problem in a structural causal model; use sparse autoencoder latents of the PRM to find features that separate hacked from normal steps, then intervene on those latents over a prior distribution and average the PRM outputs (backdoor adjustment) to get a debiased step reward without changing the policy or retraining the PRM.

Key results

• With Qwen2.5-Math-PRM-7B, CRA raises MATH accuracy for Qwen2.5-0.5B-Instruct policy from 46.6 to 48.6 and GSM8K from 60.9 to 62.3 (beam 4).
• Average MATH accuracy improves from 57.3% to 60.2% and average GSM8K from 77.9% to 79.5% across settings (beam 4).
• Causal intervention lowers PRM scores of reward-hacking steps by about 0.04 on average while random feature interventions have near-zero effect.

Surprising findings

• Reward hacking steps account for 49.5% of error cases in the authors' experiments.

Limitations

• Backdoor adjustment assumes a balanced or empirical prior over a single identified confounder feature; the causal assumptions are not empirically validated.
• Reward-hacking labels are produced by manual labeling and depend on the chosen t-statistic and activation thresholds.
• Gains are modest (about 1-3 points) and inconsistent across settings, e.g. Math-Shepherd on GSM8K with Qwen2.5-Math policy shows no gain.

Experimental setup

Datasets: GSM8K MATH

Benchmarks: GSM8K MATH

Metrics: final answer accuracy under beam search (beam 4 and beam 2) PRM score change on reward-hacking vs normal steps

Baselines: Math-Shepherd-PRM-7B Qwen2.5-Math-PRM-7B

Builds on: Math-Shepherd Qwen2.5-Math-PRM sparse autoencoders beam search

Closest papers by meaning

• Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning (ICLR 2026 · similarity 0.89)
• Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning (NeurIPS 2025 · similarity 0.88)
• Unlocking Multimodal Mathematical Reasoning via Process Reward Model (NeurIPS 2025 · similarity 0.86)
• Adversarial Training for Process Reward Models (ICML 2026 · similarity 0.86)
• Rethinking Reward Models for Multi-Domain Test-Time Scaling (NeurIPS 2026 · similarity 0.86)
• Self-Aligned Reward: Towards Effective and Efficient Reasoners (ICLR 2026 · similarity 0.86)