Research map / Reinforcement learning / Policy gradient and actor-critic

Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation

NeurIPS 2024 · Reinforcement learning › Policy gradient and actor-critic · approach: Importance-weighted OPE estimators · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes STAR, an off-policy evaluation framework that estimates small tabular abstract reward processes (ARPs) from importance-weighted data via state abstraction and weight clipping, yielding consistent estimators that outperform baselines on CartPole, ICU-Sepsis, and Asterix.

Problem

Off-policy evaluation of policies from logged data suffers from high variance in importance sampling methods and irreducible bias in model-based methods due to model class mismatch.

Approach

Map states to a finite set of abstract states via clustering (CluSTAR), fit a tabular abstract reward process using importance-weighted maximum likelihood, and clip weights to the c most recent ratios; the resulting ARP return is consistent and the framework subsumes WPDIS and approximate-model estimators as special cases.

Key results

• The best STAR estimator outperforms all baselines in all twelve studied cases, and the median STAR estimator surpasses baselines in seven of twelve.
• On ICU-Sepsis, the best ARP estimators achieve prediction errors roughly an order of magnitude lower than the best baseline.
• Best ARP configurations were |Z|=32 for CartPole, |Z|=16 for ICU-Sepsis, and |Z|=8 for Asterix.

Measured effects reported in the paper

Surprising findings

• The framework is not a single estimator; the best configuration varies with dataset size and domain, and the authors note that the naive clustering-based abstraction is competitive without automated abstraction discovery.

Limitations

• Requires knowledge of behavior policy action probabilities for importance weights.
• No principled method for selecting the abstraction function and weight clipping factor.
• Clipping introduces asymptotic bias unless the abstraction is c-th order Markov, which is not verified in practice.

Experimental setup

Datasets: CartPole ICU-Sepsis MinAtar Asterix

Benchmarks: CartPole ICU-Sepsis MinAtar Asterix

Metrics: mean squared prediction error

Baselines: Vanilla IS Per-Decision IS Weighted IS WPDIS MBased FQE MRDR DR MAGIC

Builds on: weighted per-decision importance sampling approximate model estimator k-means clustering Caltech OPE Benchmarking Suite

Closest papers by meaning

• Combining parametric and nonparametric models for off-policy evaluation (ICML 2019 · similarity 0.89)
• More Efficient Off-Policy Evaluation through Regularized Targeted Learning (ICML 2019 · similarity 0.89)
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (NeurIPS 2021 · similarity 0.89)
• Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol (NeurIPS 2025 · similarity 0.88)
• Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation (ICML 2025 · similarity 0.88)
• Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design (ICML 2024 · similarity 0.88)