Research map / Reinforcement learning / MDPs and planning
Demonstration-Regularized RL
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically that demonstration-regularized RL (behavior cloning followed by KL-regularized planning) and its RLHF variant achieve sample complexity scaling as 1/N_exp in finite and linear MDPs, with tight behavior cloning KL bounds.
Problem
How much do N expert demonstrations reduce the sample complexity of best policy identification (BPI) in finite and linear MDPs, both for RL with rewards and for preference-based RLHF?
Approach
Reduce BPI with demonstrations to KL-regularized BPI around a behavior cloning policy; new optimistic solvers (UCBVI-EntP, LSVI-Ent) exploit strong convexity of the KL regularizer to obtain 1/(lambda*eps) sample complexity, and a KL-regularized RLHF variant avoids pessimism by collecting preferences with the BC policy.
Key results
- Demonstration-regularized RL achieves sample complexity O~(H^6 S^3 A^2/(N_exp eps^2)) in finite MDPs and O~(H^6 d^3/(N_exp eps^2)) in linear MDPs.
- Behavior cloning with KL regularization gives trajectory KL error O~(SAH/N_exp) in finite MDPs and O~(dH/N_exp) in linear MDPs.
- UCBVI-Ent-P achieves O~(H^5 S^2 A/(lambda eps)) regularized BPI sample complexity, improving on the prior O~(H^8 S^4 A/(lambda eps)).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use expert demonstrations to learn a behavior-cloning reference policy for regularized BPI | BPI in finite MDPs · Finite MDP | sample complexity: \tcO(\mathrm{Poly}(S,A,H)/\epsilon^2) -> \tcO(\mathrm{Poly}(S,A,H)/(\epsilon^2 \Nexp)) | Helped |
| Use expert demonstrations to learn a behavior-cloning reference policy for regularized BPI | BPI in linear MDPs · Linear MDP | sample complexity: \tcO(\mathrm{Poly}(d,H)/\epsilon^2) -> \tcO(\mathrm{Poly}(d,H)/(\epsilon^2 \Nexp)) | Helped |
| Use UCBVIEntp instead of UCBVIEnt | Demonstration-regularized RL · Regularized BPI | sample complexity: \tcO(1/\varepsilon^2) -> \tcO(1/(\varepsilon\lambda)) | Helped |
| Use UCBVIEntp instead of the previous fast-rate finite-MDP algorithm | Regularized BPI in finite MDPs · Finite MDP | sample complexity: \tcO(H^8 S^4 A / (\lambda \varepsilon)) -> \tcO\left( \frac{H^5 S^2 A}{ \lambda \varepsilon} \right) | Helped |
| Collect preferences offline using the behavior-cloning policy and regularize RLHF toward that policy | Preference-based BPI with demonstrations · Finite and linear MDPs | sample complexity: \tcO\left( \frac{\mathrm{Poly}(S,A,H)}{\epsilon^2 \Nexp} \right); \tcO\left( \frac{\mathrm{Poly}(d,H)}{\epsilon^2 \Nexp} | Helped |
| Use behavior cloning alone instead of reward-model fine-tuning | Demonstration-regularized RLHF with a deterministic expert · Preference-based BPI with demonstrations | policy optimality | No clear effect |
Surprising findings
- Using UCBVI-Ent (prior algorithm) within demonstration-regularized RL yields no acceleration from demonstrations due to its 1/eps^2 rate.
- Demonstration-regularized RLHF avoids explicit pessimism and matches the sample complexity rates of RL with demonstrations.
Limitations
- Requires expert policy near optimal and a well-specified policy class (covering and minimal-probability assumptions).
- Linear setting requires log-linear expert parametrization.
- Results are purely theoretical with no empirical validation.
Experimental setup
Metrics: sample complexity KL divergence suboptimality gap
Baselines: UCBVI-Ent RFExploreEnt behavior cloning
Builds on: UCBVI-Ent KL-regularized MDP Bradley-Terry-Luce preference model
Closest papers by meaning
- Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations (NeurIPS 2025 · similarity 0.91)
- Sharp Analysis for KL-Regularized Contextual Bandits and RLHF (NeurIPS 2025 · similarity 0.90)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective (ICML 2025 · similarity 0.89)
- Logarithmic Regret for Online KL-Regularized Reinforcement Learning (ICML 2025 · similarity 0.88)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.