Research map / Reinforcement learning / Policy gradient and actor-critic

A Unifying View of Linear Function Approximation in Off-Policy RL Through Matrix Splitting and Preconditioning

NeurIPS 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Model-based RL with guarantees · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Gives necessary and sufficient convergence conditions for linear TD in off-policy policy evaluation via matrix splitting and preconditioning, showing TD, PFQI and FQI differ by preconditioner rather than number of updates.

Problem

Off-policy policy evaluation with linear function approximation: when does TD converge, and how do TD, partial fitted Q-iteration and FQI relate?

Approach

Views TD as an iterative solver (matrix splitting with a positive-constant preconditioner) for the target linear system, deriving exact convergence conditions (consistency, positive semi-stability, index ≤ 1, step-size bound) and an encoder-decoder interpretation of the system dynamics matrix.

Key results

• TD converges for any initial point if and only if the target linear system is consistent and the iteration matrix I - alpha(Sigma_cov - gamma Sigma_cr) is semiconvergent.
• For full-column-rank or rank-invariant features, TD is stable iff Sigma_cov - gamma Sigma_cr is positive semi-stable with index at most 1.
• TD is stable when state-action feature vectors are orthogonal, and in the on-policy setting when Phi is not full column rank.

Surprising findings

• The commonly held view that TD, PFQI and FQI differ only in the number of updates toward the target is claimed to be inaccurate.
• The star MDP and prior batch-setting analyses are said to show TD can diverge with over-parameterized linearly independent features, and one prior sufficient condition is claimed to be incorrect.

Limitations

• Analysis is restricted to linear function approximation and expected (and extended batch/stochastic) TD.
• Batch TD extension is argued by replacing matrices with empirical counterparts rather than fully proven in the excerpt.

Experimental setup

Baselines: TD FQI PFQI

Builds on: temporal difference learning fitted Q-iteration partial fitted Q-iteration

Closest papers by meaning

• Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
• Almost Sure Convergence of Linear Temporal Difference Learning with Arbitrary Features (NeurIPS 2026 · similarity 0.89)
• Horizon Adaptive Offline Policy Learning via Value Stitching (NeurIPS 2026 · similarity 0.88)
• Revisiting a Design Choice in Gradient Temporal Difference Learning (ICLR 2025 · similarity 0.88)
• Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation (ICML 2024 · similarity 0.88)
• Finite Sample Analysis of Linear Temporal Difference Learning with Arbitrary Features (NeurIPS 2025 · similarity 0.88)