Research map / Reinforcement learning / RL from feedback

S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: Theory of RLHF · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes S2T-RLHF, which decomposes a sequence-level preference reward into sentence-level allocations via Nash-style bargaining and then into token-level weights via a Dirichlet allocation network, improving PPO training stability without reward-model retraining.

Problem

Standard RLHF with a single sequence-level scalar reward leaves within-response credit assignment ambiguous, causing unstable training; finer token-level reward refinement may amplify reward noise.

Approach

Allocate the sequence reward across sentences with a Nash-bargaining-style fixed-point over semantic influence vectors (estimated by deletion perturbations and ridge regression), then refine within each sentence with a bounded Dirichlet token allocation (DTAN) that modulates the PPO advantage; argues sentence granularity is the stable middle ground.

Key results

• Against sequence-level RLHF under a GPT-4o-mini judge, S2T-RLHF wins 53.0% vs 36.5% lose.
• Under RM-Gemma-7B judge, S2T-RLHF wins 51.1% vs 39.5% lose against RLHF.
• Ablation: S2T-RLHF beats sentence-only (Sent.) at 54.5% vs 35.4% (LLM judge) and token-only (Token) at 48.6% vs 41.6%.

Surprising findings

• Token-only refinement is more prone to early-stage instability under aggressive learning rates than sentence-level allocation.
• Fine-grained token-level decomposition methods collapse under more aggressive optimization.

Limitations

• Reward decomposition mitigates but does not eliminate RLHF instability; does not correct reward model misspecification.
• Sentence influence estimation requires multiple perturbation-based reward queries per response, adding computation.
• Results reported over limited settings (single policy/RM pair, primarily HH-RLHF).

Experimental setup

Datasets: HH-RLHF

Benchmarks: HH-RLHF AdvBench RealToxicityPrompts ToxiGen SafeRLHF

Metrics: win rate lose rate tie rate KL divergence policy entropy mean reward

Baselines: RLHF ABC SCAR TLCR RTO MA-RLHF

Builds on: PPO RLHF Nash bargaining DPO

Closest papers by meaning

• MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.90)
• Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning (AAAI 2025 · similarity 0.90)
• Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback (AAAI 2025 · similarity 0.89)
• Greedy Sampling Is Provably Efficient For RLHF (NeurIPS 2025 · similarity 0.89)
• Provably Efficient Online RLHF with One-Pass Reward Modeling (NeurIPS 2025 · similarity 0.89)
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (ICML 2024 · similarity 0.89)