Research map / Optimization / Implicit bias and generalization
Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically analyzes implicit bias of policy gradient for underdetermined linear quadratic regulator problems and shows that learned linear controllers extrapolate to unseen initial states depending on the exploration induced by the system, with supporting experiments on pendulum and quadcopter neural controllers.
Problem
Whether policy gradient's implicit bias in optimal control (underdetermined LQR) leads learned controllers to extrapolate to initial states unseen in training.
Approach
Shows extrapolation of policy-gradient-learned linear controllers depends on the exploration the system induces from training initial states; proves non-extrapolation without exploration, perfect extrapolation with shift dynamics, and non-trivial extrapolation for random systems; also shows the implicit bias is not minimum Euclidean norm.
Key results
- For shift dynamics with horizon H, policy gradient's controller has optimality and cost measure ratios bounded by 4(D-1)^2/(H+D)^2, approaching perfect extrapolation as H grows.
- For random Gaussian system matrices, a single policy gradient iteration yields non-trivial extrapolation in expectation: E[optmes(K2)]/E[optmes(Kmin)] <= 1 - eta*H(H-1)/D.
- The policy-gradient controller's squared norm exceeds the minimum-norm training-optimal controller by sum_{d=2}^D (1-2(d-1)/(H+D))^2 = Omega(D), so implicit bias is not Euclidean norm minimization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a non-exploratory system with A = B = I | Policy gradient in underdetermined LQR · Unseen initial states | optimality measure ratio vs. minimum-norm baseline: 1 | No clear effect |
| Use a shift transition matrix and train from e1, with H divisible by D | Policy gradient in underdetermined LQR · Unseen initial states | optimality measure ratio vs. minimum-norm baseline: ≤ 4(D - 1)^2 / (H + D)^2 | Helped |
| Use a shift transition matrix and train from e1, with H divisible by D | Policy gradient in underdetermined LQR · Unseen initial states | cost measure ratio vs. minimum-norm baseline: ≤ 4(D - 1)^2 / (H + D)^2 | Helped |
| Use a random Gaussian transition matrix | Policy gradient in underdetermined LQR · Unseen initial states | expected optimality measure ratio vs. minimum-norm baseline: ≤ 1 - η · H (H - 1) / D | Helped |
| Use a random Gaussian transition matrix, under the stated dimension and learning-rate conditions | Policy gradient in underdetermined LQR · Unseen initial states | optimality measure ratio vs. minimum-norm baseline: ≤ 1 - η · H (H-1) / 4D | Helped |
| Set Q so q2 = ··· = qD = 0 and q1 > 0 | Policy gradient with the shift transition matrix · Unseen initial states | Q-optimality measure: zero | Helped |
| Set Q so q2 = ··· = qD = 0 and q1 > 0 | Policy gradient with the shift transition matrix · Unseen initial states | Q-cost measure: zero | Helped |
| Train neural network controllers with policy gradient on the pendulum and quadcopter tasks | ReLU neural network controller · Pendulum and quadcopter unseen initial states lower than training states | normalized cost measure of extrapolation | Helped |
| Train neural network controllers with policy gradient on the pendulum and quadcopter tasks | ReLU neural network controller · Pendulum and quadcopter unseen initial states lower than training states | extrapolation performance | Helped |
| Replace the identity cost matrix with a diagonal Q having only q1 positive (ablation) | Policy gradient in the shift-system setting · Unseen initial states | extrapolation: zero Q-optimality and cost measures | Helped |
Surprising findings
- Policy gradient's implicit bias does not minimize the Euclidean norm among training-optimal controllers, unlike gradient descent in linear regression.
- Neural network controllers for pendulum and quadcopter extrapolate near-perfectly to unseen lower-height initial states despite non-extrapolating solutions existing.
Limitations
- Theoretical guarantees for random systems require learning rates and dimensions that grow super-exponentially with horizon ((4H-1)!!).
- Extrapolation guarantees only for the first iteration of policy gradient and only for the optimality measure, with Gaussian transition matrices only.
- Non-linear experiments are empirical; theory covers only linear controllers with unregularized controls (R=0).
Experimental setup
Benchmarks: simulated LQR problems pendulum control quadcopter control
Metrics: optimality measure ratio cost measure ratio
Baselines: minimum-norm non-extrapolating controller
Builds on: policy gradient for LQR implicit bias of gradient descent in linear regression
Closest papers by meaning
- Does “Do Differentiable Simulators Give Better Policy Gradients?” Give Better Policy Gradients? (ICLR 2026 · similarity 0.88)
- Backpropagation through the Void: Optimizing control variates for black-box gradient estimation (ICLR 2018 · similarity 0.88)
- Online Policy Gradient for Model Free Learning of Linear Quadratic Regulators with $\sqrt$T Regret (ICML 2021 · similarity 0.88)
- Global Convergence of Policy Gradient Methods for the Linear Quadratic Regulator (ICML 2018 · similarity 0.88)
- Identifying Policy Gradient Subspaces (ICLR 2024 · similarity 0.87)
- On the Second-Order Convergence of Biased Policy Gradient Algorithms (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.