Research map / Reinforcement learning / Offline RL
Probabilistic Offline Policy Ranking with Approximate Bayesian Computation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes POPR, a framework for offline policy ranking that estimates each candidate policy's performance posterior from expert data via energy-based approximate Bayesian computation (POPR-EABC), enabling worst/best-case and probabilistic comparisons without rewards.
Problem
Offline policy ranking (OPR) without access to behavior policy probabilities or reliable rewards, especially for worst-case, best-case, and probabilistic comparisons of candidate policies where OPE methods give only point estimates and fail under sparse rewards.
Approach
Characterize each candidate policy by the probability θ that it matches expert actions, and sample its posterior via ABC Metropolis-Hastings using a smooth energy function (normalized JS-based similarity between bootstrapped expert and policy trajectories) and a Beta pseudo-likelihood fitted by method of moments, so ranking uses posterior samples rather than value estimates.
Key results
- POPR-EABC reaches NDCG and SRCC of 1.0000 on ToyEnv and Acrobot for multi-level differentiable policies, and 0.9992 NDCG / 0.9663 SRCC on MountainCar, outperforming all baselines.
- In best/worst/mean-case ranking on MountainCar, POPR-EABC gives the highest NDCG and SRCC across all cases, while baselines like FQE and BayesDICE fail under worst-case analysis.
- POPR-EABC produces smaller standard deviations than baselines in the best/worst-case experiments, which the authors attribute to parameterizing energy variance in the pseudo-likelihood.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use POPR-EABC instead of direct action-agreement ranking (ablation) | POPR · MountainCar | NDCG: 0.9829 -> 0.9992 | Helped |
| use POPR-EABC instead of direct action-agreement ranking (ablation) | POPR · MountainCar | SRCC: 0.9357 -> 0.9663 | Helped |
| use POPR-EABC instead of direct action-agreement ranking (ablation) | POPR · Pendulumn | NDCG: 0.9421 -> 0.9908 | Helped |
| use POPR-EABC instead of direct action-agreement ranking (ablation) | POPR · Pendulumn | SRCC: 0.8190 -> 0.9047 | Helped |
| use POPR-EABC instead of direct action-agreement ranking (ablation) | POPR · ToyEnv | NDCG: 1.0000 -> 1.0000 | No clear effect |
| use POPR-EABC instead of direct action-agreement ranking (ablation) | POPR · AcroBot | NDCG: 1.0000 -> 1.0000 | No clear effect |
Surprising findings
- Tree-Backup(λ) and AgreeRank also achieve perfect scores on ToyEnv, and FQE and other OPE baselines produce negative SRCC on several environments.
Experimental setup
Datasets: ToyEnv MountainCar Acrobot Pendulum
Benchmarks: ToyEnv MountainCar Acrobot Pendulum
Metrics: NDCG SRCC
Baselines: FQE Tree-Backup(λ) Q^π(λ) MBased BayesDICE AgreeRank
Builds on: Fitted Q-Evaluation BayesDICE Supervised Off-Policy Ranking Metropolis-Hastings
Closest papers by meaning
- Supervised Off-Policy Ranking (ICML 2022 · similarity 0.88)
- Off-Policy Evaluation for Ranking Policies under Deterministic Logging Policies (ICLR 2026 · similarity 0.88)
- OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators (NeurIPS 2024 · similarity 0.87)
- Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes (NeurIPS 2024 · similarity 0.85)
- Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation (NeurIPS 2024 · similarity 0.85)
- Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.