Research map / Reinforcement learning / RL from feedback

PoEM: Predicting New RL Outcomes from Existing Policies

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Theory of RLHF

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Predicts the outcome of RL on a new reward by composing existing single-reward post-trained adapters as a weighted product of experts in log-policy space, with weights regressed from a small calibration set and no new RL run.

Problem

Post-training with RL is expensive and must be redone for every new or combined reward; the question is whether the outcome on a new reward can be predicted from already-trained policies without running RL.

Approach

If the target reward is a linear combination of basis rewards, the optimal KL-regularized policy is a weighted log-mixture (product of experts) of the basis policies. Even for non-linear rewards, the log-ratios of trained experts span a low-rank subspace, so the target log-policy can often be approximated by a weighted combination of basis log-ratios, with a coverage score (R^2) indicating when th

Key results

• On combined programmatic rewards, PoEM with true weights reaches reward error 0.19 on P-GRPO (0.28 with fitted weights), close to a second RL run's 0.12, while best-of-16 gives 0.37.
• Log-ratio matrix of 20 programmatic experts has effective rank 6.3 versus 19.4 for weight updates, showing a policy-space rank gap.
• On held-out rewards, coverage ranks reward error with Spearman rho from -0.71 to -0.81; covered rewards recover 0.55 to 0.69 of RL's gain versus 0.22 to 0.43 for uncovered ones.
• On the RM-Div basis PoEM is closer to the held-out expert than the top expert on 9 of 10 rewards.

Surprising findings

• Log-policies of experts trained on rewards with no obvious linear relation span a low-rank subspace, even though their weight updates are nearly orthogonal.
• Held-out rewards can be recovered from other experts' log-ratios even when the reward itself is far from the span of the other rewards.

Limitations

• Coverage is bounded by the span of the chosen basis; rewards orthogonal to the available adapters cannot be recovered.
• Exact composition assumes KL-regularized optimal experts with shared reference model and beta, which often does not hold in practice.
• Tested only on small models (Qwen3-0.6B, SD v1.4); not tested on long-context generation or across reference models.

Experimental setup

Datasets: UltraChat PKU-SafeRLHF ImageNet class names

Benchmarks: held-out UltraChat prompts PKU-SafeRLHF red-team prompts held-out image prompts

Metrics: reward recovery reward error relative KL coverage R^2 effective rank

Baselines: best-of-N top expert uniform weights DeRa MOD second RL run GDPO

Builds on: DPO GRPO DDPO KL-regularized RL closed form Tweedie's formula MOD DeRa

Closest papers by meaning

• A Theoretical Framework for Partially-Observed Reward States in RLHF (ICLR 2025 · similarity 0.88)
• RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style (ICLR 2025 · similarity 0.88)
• Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback (NeurIPS 2025 · similarity 0.88)
• The Trickle-down Impact of Reward Inconsistency on RLHF (ICLR 2024 · similarity 0.88)
• Information-Theoretic Reward Decomposition for Generalizable RLHF (NeurIPS 2025 · similarity 0.88)
• Policy Filtration for RLHF to Mitigate Noise in Reward Models (ICML 2025 · similarity 0.87)