Research map / Reinforcement learning / Offline RL
Scaling Offline Evaluation of Reinforcement Learning Agents through Abstraction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
An AAAI 2024 invited talk abstract summarizing the author's work on scaling offline policy evaluation (OPE) for RL by using state abstractions, both expert-specified and learned.
Problem
Offline policy evaluation in RL is unreliable in high-dimensional or real-world domains because existing OPE methods try to adjust for all differences between the evaluation policy and the behavior data, including irrelevant ones.
Approach
Use state abstraction so OPE only adjusts for state differences relevant to policy performance: first with expert-specified irrelevant state variables, then with a learned state representation that identifies irrelevant differences from data.
Key results
- Prior work (Pavse and Hanna 2023a) showed that a given state abstraction yields more accurate OPE estimates in theory and in practice.
- Follow-up work (Pavse and Hanna 2023b) learns state representations to capture irrelevant state differences in high-dimensional RL domains when experts cannot specify them a priori.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a given state abstraction to avoid adjusting for state variables specified as irrelevant | Offline policy evaluation (OPE) · OPE | estimation accuracy | Helped |
| Learn a state representation to identify irrelevant state differences from data | Offline policy evaluation (OPE) · high-dimensional RL domains | OPE effectiveness | Helped |
Limitations
- Abstract summarizes prior work only; no experimental details or quantitative results are given.
- Current OPE methods often fail to accurately evaluate untested policies when large datasets are unavailable.
Experimental setup
Builds on: state abstraction offline policy evaluation
Closest papers by meaning
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.91)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.90)
- Horizon Reduction Makes RL Scalable (NeurIPS 2025 · similarity 0.90)
- Dispelling the Mirage of Progress in Offline MARL through Standardised Baselines and Evaluation (NeurIPS 2024 · similarity 0.90)
- The Generalization Gap in Offline Reinforcement Learning (ICLR 2024 · similarity 0.89)
- Offline Actor-Critic Reinforcement Learning Scales to Large Models (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.