Research map / Reinforcement learning / MDPs and planning

Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching

NeurIPS 2026 · Reinforcement learning › MDPs and planning · approach: Offline RL sample complexity · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Q-MMR, a top-down moment-matching method that learns one weight per data point for finite-horizon off-policy evaluation, yielding a dimension-free data-dependent bound under only Q^pi realizability.

Problem

Off-policy evaluation in finite-horizon MDPs with general function approximation: existing FQE analyses pay a log-complexity term of the function class and require Bellman completeness, and the nature of coverage in GFA is not well understood.

Approach

Learn per-data-point scalar weights level-by-level by minimizing an IPM-style moment-matching loss against a value-function discriminator class; the telescoping analysis only concentrates on Q^pi, so no union bound over the function class is needed. With linear classes it coincides with linear FQE and MC/IS in special cases.

Key results

• Data-dependent finite-sample bound for Q-MMR is dimension-free under only Q^pi realizability: error scales with the second moment of learned weights and the empirical matching loss, with no log|F| term.
• For linear function classes with invertible covariance, Q-MMR with least-second-moment weights gives exactly the same estimate as linear FQE (no ridge).
• Under Bellman completeness, the population weights satisfy ||w*_h||_{2,d^D_h} = sup_f (E_{d^pi}[f])^2/(E_{d^D}[f])^2 type coverage, which generalizes linear feature-dynamics coverage.

Surprising findings

• The analysis performs no union bound over the function class, yet yields a dimension-free bound, contrary to the common log|F| dependence of GFA OPE analyses.
• Coverage target is the feature-dynamics vector from an approximate linear system rather than the true feature occupancy, which is generally different without Bellman completeness.

Limitations

• Requires i.i.d. trajectory data; adaptively collected data breaks the required conditional independence.
• Weights are chosen greedily level-by-level and ignore effects on downstream levels; joint optimization runs into double-sampling issues.
• Without Bellman completeness, weights may blow up exponentially in horizon as with linear FQE; the data-independent bound incurs dimension dependence in higher-order terms.

Experimental setup

Baselines: FQE linear FQE Monte-Carlo estimator step-wise importance sampling MWL

Builds on: fitted Q evaluation marginalized importance sampling linear FQE MWL

Closest papers by meaning

• Minimax Weight and Q-Function Learning for Off-Policy Evaluation (ICML 2020 · similarity 0.89)
• RL in Latent MDPs is Tractable: Online Guarantees via Off-Policy Evaluation (NeurIPS 2024 · similarity 0.87)
• On the Complexity of Offline Reinforcement Learning with Q*-Approximation and Partial Coverage (NeurIPS 2026 · similarity 0.86)
• Optimal Single-Policy Sample Complexity and Transient Coverage for Average-Reward Offline RL (NeurIPS 2025 · similarity 0.86)
• A Unifying View of Linear Function Approximation in Off-Policy RL Through Matrix Splitting and Preconditioning (NeurIPS 2025 · similarity 0.86)
• Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes (NeurIPS 2024 · similarity 0.86)