Research map / Reinforcement learning / Policy gradient and actor-critic
Impact of Computation in Integral Reinforcement Learning for Continuous-Time Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that quadrature error in the policy-evaluation integral of integral reinforcement learning (IntRL) degrades policy-iteration convergence, frames PI as Newton's method on the HJB equation with bounded error, and proves O(N^-2) (trapezoidal) and O(N^-b) (Bayesian quadrature with Matérn kernel) local convergence rates validated on two simulated control tasks.
Problem
Integral reinforcement learning for continuous-time control requires computing the integral of the utility function from discrete state samples; quadrature errors in this policy-evaluation step can affect the convergence and final quality of the learned controller, which has not been studied.
Approach
Interprets IntRL policy iteration as Newton's method on the HJB equation so that quadrature error becomes a bounded extra error per iteration; shows Bayesian quadrature with the RKHS-inducing (Matérn) kernel is worst-case optimal, giving provable convergence rates in the number of samples.
Key results
- Empirically, parameter error of the learned value function decays as O(N^-2) with trapezoidal quadrature and O(N^-4) with Bayesian quadrature using a Matérn kernel (b=4) on the linear example.
- On the nonlinear example, the same rates O(N^-2) (trapezoidal) and O(N^-4) (BQ Matérn, b=4) are reported for learned parameters.
- Theory gives local convergence rates O(N^-2) for trapezoidal rule and O(N^-b) for BQ with Matérn kernel of smoothness b in IntRL policy iteration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BQ with a Matérn kernel (b=4) instead of the trapezoidal rule for PEV integration | IntRL · Linear system (Example 1) | learned parameter error convergence rate: O(N^{-2}) -> O(N^{-4}) | Helped |
| Use BQ with a Matérn kernel (b=4) instead of the trapezoidal rule for PEV integration | IntRL · Nonlinear system (Example 2) | learned parameter error convergence rate: O(N^{-2}) -> O(N^{-4}) | Helped |
| Use an adaptive Runge-Kutta (4,5) solver instead of the Euler method | CTRL with known internal dynamics · Cartpole | accumulated cost | Helped |
| Reduce the Euler method step size | CTRL with known internal dynamics · Cartpole | accumulated cost | Helped |
Surprising findings
- Computation in the policy-evaluation integral impacts control performance even when the system dynamics are known, as shown with Euler vs Runge-Kutta solvers on CartPole in the appendix.
Limitations
- Analysis is local and relies on a Banach-space Newton framework with strong smoothness and boundedness conditions.
- Value function approximation learning error is assumed negligible.
- Experiments limited to two low-dimensional simulated systems (third-order linear, two-state nonlinear).
Experimental setup
Benchmarks: linear-quadratic regulator third-order system canonical nonlinear control system (IntRL example)
Metrics: parameter error norm vs sample size N control gain Frobenius error average accumulated cost difference average control difference
Baselines: trapezoidal rule
Builds on: integral reinforcement learning Bayesian quadrature Newton's method error analysis policy iteration
Closest papers by meaning
- Alternating Optimisation and Quadrature for Robust Control (AAAI 2018 · similarity 0.86)
- Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States (ICML 2024 · similarity 0.84)
- Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network (ICML 2025 · similarity 0.84)
- A Unifying View of Linear Function Approximation in Off-Policy RL Through Matrix Splitting and Preconditioning (NeurIPS 2025 · similarity 0.84)
- Sample Efficient Reinforcement Learning with Partial Dynamics Knowledge (AAAI 2024 · similarity 0.84)
- Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.