Research map / Reinforcement learning / MDPs and planning
Model-Free Robust $\phi$-Divergence Reinforcement Learning Using Both Offline and Online Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RPQ, a model-free robust phi-regularized fitted Q-iteration for offline data, and HyTQ, a hybrid offline+online total-variation-regularized Q-iteration, with suboptimality guarantees under general function approximation.
Problem
Learning robust policies for MDPs under phi-divergence parameter uncertainty (RRMDP framework) with large state spaces and general function approximation, using offline data alone or combined offline and online data.
Approach
Replace the inner convex minimization of the robust regularized Bellman operator with a dual-variable function minimization over a function class, then run fitted Q-iteration with empirical risk minimization for the dual variable and least squares for Q; extend to hybrid RL by aggregating offline and on-policy datasets with backward induction, using a bilinear-model and robust Bellman error transf
Key results
- RPQ achieves suboptimality of order lambda sqrt(C log(|F||G|)) / ((1-gamma)^3 sqrt(N)) for TV divergence under robust exploratory offline data.
- HyTQ offers cumulative suboptimality under single-policy concentrability plus robust Bellman error transfer coefficient, with bound scaling as sqrt(d H^3)(lambda+H)/sqrt(N) for offline+online data.
- Theoretical comparison: in the tabular setting, RPQ with lambda=1/(1-gamma) gives |S||A|/((1-gamma)^3 sqrt(N)) versus N^{-1/3} for prior stochastic-approximation Q-learning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RPQ for offline robust learning with general φ-divergences | Offline robust RL · Offline robust RL, general φ-divergence | suboptimality bound: $N^{-1/2}$ vs. $N^{-1/3}$ | Helped |
| Use RPQ instead of prior robust Q-learning | Offline χ-square robust RL, tabular · Offline χ-square robust RL, tabular | suboptimality bound: $\widetilde{\mathcal{O}}\left(\frac{ |\mathcal{S}||\mathcal{A}| }{ (1-\gamma)^{3} \sqrt{N} }\right)$ vs. $\widetilde{\ma | Helped |
| Combine offline data with online interaction | HyTQ · Hybrid robust RL, total variation | offline data coverage requirement | Helped |
| Use HyTQ with offline and online data | Hybrid robust RL · Finite-horizon hybrid robust RL, total variation | cumulative suboptimality | Helped |
Surprising findings
- Theoretical only: the KL-specific Q-update has an exponential dependence on horizon when extending the TV-style analysis to other phi-divergences, forcing a TV-only hybrid algorithm.
Limitations
- Suboptimality bounds are worse than model-based robust value iteration by factors of sqrt(|S||A|) and 1/(1-gamma); no matching lower bound.
- HyTQ is proposed only for total variation divergence; general phi-divergence hybrid case has exponential horizon dependence.
- Requires strong assumptions: robust exploratory coverage, approximate Bellman completeness, bilinear models, and fail-state.
Experimental setup
Metrics: suboptimality bound sample complexity
Baselines: Robust fitted Q-iteration for TV divergence Robust Q-iteration for KL divergence Robust Q-learning for phi-divergence (tabular) Hybrid Q-learning (HyQ)
Builds on: fitted Q-iteration hybrid Q-learning robust Markov decision process penalized distributionally robust optimization
Closest papers by meaning
- Robust Offline Reinforcement Learning with Linearly Structured $f$-Divergence Regularization (ICML 2025 · similarity 0.93)
- Time-Constrained Robust MDPs (NeurIPS 2024 · similarity 0.89)
- Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithms (NeurIPS 2024 · similarity 0.88)
- Sample Complexity of Distributionally Robust Average-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- State Entropy Regularization for Robust Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.