Research map / Reinforcement learning / Visual and representation RL
Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COTD, a conditional VAE ensemble that models RL transition dynamics and uses inductive conformal prediction on reconstruction error to detect state- and environment-based OOD execution with a guaranteed true positive rate.
Problem
Detecting out-of-distribution execution of deep RL agents, which lacks a consensus definition and existing detectors cannot cover all causes (state uncertainty, transition shifts, perturbations) or give probabilistic guarantees.
Approach
Define OOD execution via the empirical transition likelihood, learn transitions with a CVAE ensemble conditioned on the prior state, and apply inductive conformal prediction on reconstruction errors so that a chosen true-positive OOD detection rate is guaranteed.
Key results
- COTD achieves Overall AUC 0.807 on LunarLander versus 0.761 for UBOOD (best baseline) and 0.875 on Ant versus 0.799 for PEDM.
- On Env. AUC, COTD outperforms the best baseline by at least 17% on LunarLander, 4% on Pendulum and 8% on CartPole, per the authors.
- Ensemble size of 5 CVAEs was chosen by optimizing false positive rate; larger ensembles gave diminishing gains and slower detection.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COTD instead of the strongest listed baseline | OOD detection · LunarLander | Overall AUC: 0.761 -> 0.807 | Helped |
| Use COTD instead of the strongest listed baseline | OOD detection · Ant | Overall AUC: 0.799 -> 0.875 | Helped |
| Use COTD instead of the strongest listed baseline | OOD detection · Pendulum | Overall AUC: 0.753 -> 0.765 | Helped |
| Use COTD instead of the strongest listed baseline | OOD detection · CartPole | Overall AUC: 0.725 -> 0.759 | Helped |
| Use COTD instead of the strongest listed baseline | OOD detection · LunarLander | State AUC: 0.850 -> 0.852 | Helped |
| Use COTD instead of the strongest listed baseline | OOD detection · Ant | State AUC: 0.877 -> 0.936 | Helped |
| Use COTD instead of the strongest listed baseline | OOD detection · Pendulum | State AUC: 0.823 -> 0.797 | Hurt |
| Use COTD instead of the strongest listed baseline | OOD detection · CartPole | State AUC: 0.755 -> 0.771 | Helped |
| Use COTD instead of the strongest listed baseline | OOD detection · LunarLander | Env. AUC: 0.657 -> 0.762 | Helped |
| Use COTD instead of the strongest listed baseline | OOD detection · Ant | Env. AUC: 0.799 -> 0.814 | Helped |
| Use COTD instead of the strongest listed baseline | OOD detection · Pendulum | Env. AUC: 0.706 -> 0.733 | Helped |
| Use COTD instead of the strongest listed baseline | OOD detection · CartPole | Env. AUC: 0.706 -> 0.747 | Helped |
Surprising findings
- UBOOD achieves higher State AUC than COTD on Pendulum (0.823 vs 0.797), and DEXTER outperforms COTD on Env. (2) AUC for Ant and Pendulum.
Limitations
- Learned transition distributions are limited to Gaussian latent approximations.
- Conformal threshold guarantees true positive rate at the cost of a relatively high false positive rate.
- Detection uses transition pairs with a lag, trading detection speed for accuracy.
Experimental setup
Datasets: OpenAI Gym LunarLander OpenAI Gym Ant OpenAI Gym Pendulum OpenAI Gym CartPole
Benchmarks: LunarLander Ant Pendulum CartPole
Metrics: AUC
Baselines: PEDM DEXTER MC-DC UBOOD
Builds on: Inductive conformal prediction Conditional variational autoencoder Advantage Actor-Critic
Closest papers by meaning
- Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression (NeurIPS 2024 · similarity 0.89)
- Generative OOD-regularized Model-based Policy Optimization (NeurIPS 2026 · similarity 0.88)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.86)
- Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood (ICLR 2025 · similarity 0.85)
- Fast Decision Boundary based Out-of-Distribution Detector (ICML 2024 · similarity 0.85)
- Dynamic Uncertainty Estimation for Offline Reinforcement Learning (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.