Research map / Reinforcement learning / Policy gradient and actor-critic
Kernel Metric Learning for In-Sample Off-Policy Evaluation of Deterministic RL Policies
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Enables in-sample off-policy evaluation of deterministic continuous-action policies by kernel-relaxing the target policy and learning bandwidth and Mahalanobis metric that minimize TD update vector MSE (KMIFQE).
Problem
Off-policy evaluation of deterministic target policies in continuous action spaces, where importance-ratio-based in-sample learning fails because IS ratios are almost surely zero.
Approach
Relax the Dirac-delta target policy with a Gaussian kernel so importance resampling is well-defined, then derive the bias/variance of the TD update vector estimate and closed-form optimal bandwidth and per-state Mahalanobis metric that reduce the bias dominating in high-dimensional action spaces.
Key results
- On the Pendulum with dummy dimension, KMIFQE with learned metric achieves RMSE 0.115 vs 0.250 without metric, 1.641 for SR-DICE and 8.126 for FQE.
- KMIFQE achieves the lowest RMSE on 4 of 5 MuJoCo tasks with known behavior policy (e.g., Hopper-v2 0.023 vs FQE 0.083), with FQE best on HalfCheetah-v2 (1.637 vs 2.080 for KMIFQE).
- On D4RL medium-expert/medium-replay datasets, KMIFQE outperforms baselines on most tasks except halfcheetah-medium-expert-v2 where FQE is best (0.080 vs 0.418).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add learned kernel metric (ablation) | KMIFQE · Pendulum with dummy dim | RMSE: 0.250 $\pm$ 0.032 -> 0.115 $\pm$ 0.018 | Helped |
| add learned kernel metric (ablation) | KMIFQE · Hopper-v2 | RMSE: 0.034 $\pm$ 0.009 -> 0.023 $\pm$ 0.006 | Helped |
| add learned kernel metric (ablation) | KMIFQE · HalfCheetah-v2 | RMSE: 2.549 $\pm$ 0.017 -> 2.080 $\pm$ 0.010 | Helped |
| add learned kernel metric (ablation) | KMIFQE · Walker2d-v2 | RMSE: 0.048 $\pm$ 0.009 -> 0.032 $\pm$ 0.008 | Helped |
| add learned kernel metric (ablation) | KMIFQE · Ant-v2 | RMSE: 2.255 $\pm$ 0.014 -> 1.800 $\pm$ 0.013 | Helped |
| add learned kernel metric (ablation) | KMIFQE · Humanoid-v2 | RMSE: 0.293 $\pm$ 0.021 -> 0.246 $\pm$ 0.010 | Helped |
| add learned kernel metric (ablation) | KMIFQE · hopper-m-e-v2 | RMSE: 0.020 $\pm$ 0.005 -> 0.019 $\pm$ 0.003 | Helped |
| add learned kernel metric (ablation) | KMIFQE · halfcheetah-m-e-v2 | RMSE: 0.457 $\pm$ 0.007 -> 0.418 $\pm$ 0.016 | Helped |
| add learned kernel metric (ablation) | KMIFQE · walker2d-m-e-v2 | RMSE: 0.038 $\pm$ 0.006 -> 0.036 $\pm$ 0.006 | Helped |
| add learned kernel metric (ablation) | KMIFQE · hopper-m-r-v2 | RMSE: 0.517 $\pm$ 0.120 -> 0.536 $\pm$ 0.099 | Hurt |
| add learned kernel metric (ablation) | KMIFQE · halfcheetah-m-r-v2 | RMSE: 4.765 $\pm$ 0.026 -> 4.698 $\pm$ 0.044 | Helped |
| add learned kernel metric (ablation) | KMIFQE · walker2d-m-r-v2 | RMSE: 1.360 $\pm$ 0.025 -> 1.364 $\pm$ 0.052 | Hurt |
Surprising findings
- FQE outperforms KMIFQE on HalfCheetah-v2, which the authors attribute to slowly changing Q-values in that environment inducing low extrapolation error.
- Authors report that in high-dimensional action spaces the squared bias dominates the MSE at the optimal bandwidth, motivating metric learning.
Limitations
- Assumes a known behavior policy; unknown behavior policies require MLE estimation of the behavior policy.
- Requires the behavior policy support to cover the kernel relaxation support, and bounded IS ratios via clipping in [1e-3, 2].
- Fails to outperform FQE on HalfCheetah tasks.
Experimental setup
Datasets: Pendulum-v0 (modified with dummy action dimensions) MuJoCo (Hopper-v2, HalfCheetah-v2, Walker2d-v2, Ant-v2, Humanoid-v2) D4RL hopper/halfcheetah/walker2d medium-expert and medium-replay v2
Benchmarks: Pendulum-v0 with dummy action dimensions MuJoCo Hopper-v2, HalfCheetah-v2, Walker2d-v2, Ant-v2, Humanoid-v2 D4RL medium-expert and medium-replay datasets
Metrics: RMSE bias variance
Baselines: FQE SR-DICE KMIFQE without metric learning
Builds on: fitted Q evaluation in-sample TD learning SR-DICE kernel metric learning for contextual bandit OPE
Closest papers by meaning
- Local Metric Learning for Off-Policy Evaluation in Contextual Bandits with Continuous Actions (NeurIPS 2022 · similarity 0.92)
- Scaling Marginalized Importance Sampling to High-Dimensional State-Spaces via State Abstraction (AAAI 2023 · similarity 0.91)
- Towards Optimal Off-Policy Evaluation for Reinforcement Learning with Marginalized Importance Sampling (NeurIPS 2019 · similarity 0.91)
- Demystifying the Paradox of Importance Sampling with an Estimated History-Dependent Behavior Policy in Off-Policy Evaluation (ICML 2025 · similarity 0.89)
- Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol (NeurIPS 2025 · similarity 0.89)
- Efficient Off-Policy Learning for High-Dimensional Action Spaces (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.