Research map / Reinforcement learning / Offline RL

When Policies Cannot Be Retrained: A Unified Closed-Form View of Post-Training Steering in Offline Reinforcement Learning

NeurIPS 2026 · Reinforcement learning › Offline RL · approach: Offline RL sample complexity theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that PoE composition of a frozen offline actor with a goal-conditioned prior equals a KL-regularized adaptation rule (beta = alpha/(1-alpha)), and empirically characterizes graceful degradation under degraded priors and an actor-competence ceiling on D4RL/AntMaze.

Problem

Deployment-time adaptation of a frozen offline RL actor to a changed goal/objective, without retraining the actor or accessing new data.

Approach

Compose the frozen actor with a goal-conditioned Gaussian prior via Product-of-Experts (precision-weighted), and make explicit that this closed form coincides with a KL-regularized frozen-actor update under beta=alpha/(1-alpha), with covariances differing by a global scalar that cancels under deterministic action selection.

Key results

• Under random prior, PoE/KL-Reg achieves goal-weighted return 1499.8 vs Frozen 1410.5, Additive 164.2, Prior Only 86.3.
• Verdict split on four main-body D4RL envs (12 env-goal cells) is 4 Help / 5 Frozen / 3 Hurt.
• On six harder cells, medium-expert is Hurt in all 9 env-goal cells at every alpha, while medium-replay has 7 of 9 Frozen ties at alpha=0.9.

Surprising findings

• Medium-expert composition is Hurt in all 9 cells at every alpha, including alpha=0.9 (refined policy 90% frozen actor).
• A BC frozen actor solves zero AntMaze episodes, so every composition rule inherits zero success.

Limitations

• Theoretical bounds are loose at gamma=0.99 (penalty coefficient 19,800) and not predictive.
• Composition does not universally improve over the frozen actor; Hurt in several cells, and an actor-competence ceiling on medium-expert.
• Small evaluation budget of 5 seeds x 5 episodes per cell; AntMaze is diagnostic only.

Experimental setup

Datasets: D4RL halfcheetah-medium-v2 D4RL hopper-medium-v2 D4RL walker2d-medium-v2 D4RL hopper-medium-expert-v2 D4RL halfcheetah-medium-replay-v2 D4RL hopper-medium-replay-v2 D4RL walker2d-medium-replay-v2 D4RL halfcheetah-medium-expert-v2 D4RL walker2d-medium-expert-v2 D4RL antmaze-umaze-v2 D4RL antmaze-medium-play-v2

Benchmarks: D4RL halfcheetah-medium-v2 D4RL hopper-medium-v2 D4RL walker2d-medium-v2 D4RL hopper-medium-expert-v2 D4RL halfcheetah-medium-replay-v2 D4RL walker2d-medium-replay-v2 D4RL halfcheetah-medium-expert-v2 D4RL walker2d-medium-expert-v2 D4RL antmaze-umaze-v2 D4RL antmaze-medium-play-v2

Metrics: goal-weighted return D4RL-normalized return KL divergence along rollout states probability of improvement Help/Frozen/Hurt verdict counts

Baselines: Frozen Actor Prior Only Additive Mix KL-Reg AWR/AWAC critic-gradient step CQL-guided IQL-guided SF/GPI

Builds on: Product-of-Experts AWR/AWAC KL-regularized policy update Successor features and generalized policy improvement

Closest papers by meaning

• From Static Policies to Adaptive Priors in Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
• Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL (ICML 2024 · similarity 0.89)
• Learning to Reuse Policies in State Evolvable Environments (ICML 2025 · similarity 0.87)
• Forecasting in Offline Reinforcement Learning for Non-stationary Environments (NeurIPS 2025 · similarity 0.87)
• Offline Data Enhanced On-Policy Policy Gradient with Provable Guarantees (ICLR 2024 · similarity 0.87)
• Offline Actor-Critic Reinforcement Learning Scales to Large Models (ICML 2024 · similarity 0.87)