Research map / Reinforcement learning / MDPs and planning
Reinforcement Learning with Imperfect Transition Predictions: A Bellman-Jensen Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Bayesian value function and Bellman-Jensen gap analysis for MDPs with multi-step (imperfect) transition predictions, plus BOLA, a two-stage model-based algorithm with sample-complexity guarantees, validated on synthetic MDPs and a wind-farm storage task.
Problem
Standard RL assumes one-step transition models, but many applications have access to multi-step predictions of future states; naively augmenting the state with these predictions causes exponential blow-up, and existing theory cannot quantify the value of inaccurate or partially action-covered predic
Approach
Define a Bayesian value function over the original state space by taking expectation over K-step prediction distributions, giving a Bellman equation with a gamma^K contraction; the value of predictions is characterized by a Bellman-Jensen gap arising from reordering max and expectation operators; BOLA learns this value offline and adapts online to real-time predictions with short-horizon planning.
Key results
- On the wind farm storage task, BOLA's cost reduction over the no-prediction MDP policy improves consistently as prediction horizon K grows from 1 to 4, with the largest marginal gain at K=1.
- Prediction-based policies still outperform the no-prediction baseline at 30% relative prediction error.
- Theory: the Bellman operator for the Bayesian value function is a contraction with factor gamma^K, and BOLA's oracle sample requirement falls to (1-gamma)^-2 when K is on the order of log(1/gamma).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase prediction horizon from K=1 to K=4 | BOLA · Wind Farm Storage Control | cumulative cost reduction | Helped |
| increase relative prediction error | BOLA prediction-based policies · Wind Farm Storage Control | cost savings | Hurt |
| use longer prediction horizons under prediction noise | BOLA · Wind Farm Storage Control | cost savings | Mixed |
| use transition predictions rather than no-prediction baseline | BOLA prediction-based policies · Wind Farm Storage Control, 30% relative prediction error | cost savings | Helped |
| increase the prediction horizon when learning the Bayesian value function | BOLA · MDPs with transition predictions | prediction-oracle sample complexity | Helped |
| use BOLA with comprehensive predictions and a sufficiently long horizon instead of classical model-based RL | BOLA · MDPs with transition predictions | sample complexity | Helped |
Limitations
- Analysis is restricted to fixed-horizon planning; receding-horizon control is left for future work.
- Sample complexity bounds may not be tight and lower bounds are open.
- Analysis is for tabular finite MDPs with a generative model; the extension to model-free settings is left open.
Experimental setup
Datasets: synthetic MDPs wind farm storage control simulation
Benchmarks: wind farm storage control task
Metrics: cumulative cost reduction
Baselines: standard MDP policy without predictions
Builds on: value iteration Bellman optimality equation generative model sampling
Closest papers by meaning
- Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
- Forecasting in Offline Reinforcement Learning for Non-stationary Environments (NeurIPS 2025 · similarity 0.87)
- Inverting the Bellman Equation: From $Q$-Values to World Models (NeurIPS 2026 · similarity 0.87)
- Offline Transition Modeling via Contrastive Energy Learning (ICML 2024 · similarity 0.87)
- MetaCURL: Non-stationary Concave Utility Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
- Pausing Policy Learning in Non-stationary Reinforcement Learning (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.