Research map / Reinforcement learning / Offline RL
Stable Offline Value Function Learning with Bisimulation-based Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KROPE, a bisimulation-based kernel representation learning method that shapes state-action features so that linear LSPE value learning is provably stable and Bellman complete, and shows lower offline value error than baselines on Garnet, DMC, and D4RL.
Problem
Offline value function learning (off-policy evaluation with linear TD-style methods) can diverge depending on state-action representations; it is unclear whether bisimulation-based representations stabilize it.
Approach
Define a kernel over state-action pairs capturing pi-bisimilarity (reward similarity plus independently coupled similarity of next state-actions under the target policy), parameterize it with dot products of learned features, and prove the resulting representations give a spectral radius below 1 for LSPE and Bellman completeness under reward injectivity.
Key results
- KROPE gives the lowest error among baselines on 10 of 13 offline datasets in the LSPE linear evaluation, e.g., 0.0 on CartPoleSwingUp and CheetahRun where other methods diverge.
- In Garnet MDPs, 100% of KROPE runs have spectral radius below 1 up to latent dimension 30.
- All 100% of KROPE runs in the hyperparameter sweep yield MSVE at most 1, unlike other methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use KROPE representations rather than other auxiliary representation methods | Offline policy evaluation with LSPE · 10/13 offline datasets | stability and accuracy of normalized squared value error | Helped |
| Use KROPE instead of ROPE | Offline policy evaluation with LSPE · DMC and D4RL datasets | normalized squared value error | Helped |
| Use KROPE instead of FQE | Offline policy evaluation with LSPE · Offline datasets | stability of value estimates | Helped |
| Tune across the reported hyperparameter combinations using KROPE | Offline policy evaluation · All hyperparameter instances | fraction of runs with MSVE ≤ 1: 100% | Helped |
| Use KROPE representations rather than non-bisimulation-based representations | Garnet MDP representation learning · Garnet MDPs | correlation of representation orthogonality with action-value difference | Helped |
| Sample good transitions in the second dataset rather than bad transition pairs | KROPE representation learning · MRP divergence experiment | KROPE training loss divergence | Helped |
| Sample bad transition pairs rather than pairing a bad transition with a good transition | KROPE representation learning · MRP divergence experiment | KROPE training loss divergence | Hurt |
| Add exploration maximization of the covariance matrix | BCRL · Offline policy evaluation | normalized squared value error | Helped |
| Increase the rank of penultimate-layer features with DR3 or BEER regularization | Offline policy evaluation representations · Offline datasets | stability and accuracy | Hurt |
Surprising findings
- KROPE still diverged on 3 of 13 datasets (e.g., hopper medium, walker medium, walker med-expert), despite the stability theory.
- Sampling bad transition pairs, rather than single bad transitions, drives KROPE divergence, suggesting the auxiliary objective can be more robust than FQE.
Limitations
- KROPE uses a semi-gradient update and can still diverge in practice.
- The independent coupling introduces an additive constant that can hurt accuracy when transitions are stochastic.
- Stability results rely on the linear evaluation protocol with LSPE rather than end-to-end FQE.
Experimental setup
Datasets: Garnet MDPs DM Control CartPoleSwingUp DM Control CheetahRun DM Control FingerEasy DM Control WalkerStand D4RL cheetah D4RL hopper D4RL walker
Benchmarks: Garnet MDPs DM Control (CartPoleSwingUp, CheetahRun, FingerEasy, WalkerStand) D4RL (cheetah, hopper, walker)
Metrics: normalized squared value error (MSE) spectral radius stability fraction Pearson correlation of q-consistency performance profile
Baselines: FQE BCRL BEER DR3 DBC RoPE
Builds on: fitted q-evaluation deep bisimulation for control representations for offline policy evaluation bellman complete representation learning least-squares policy evaluation
Closest papers by meaning
- Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learning (NeurIPS 2023 · similarity 0.92)
- Representations for Stable Off-Policy Reinforcement Learning (ICML 2020 · similarity 0.90)
- Enhancing Value Function Estimation through First-Order State-Action Dynamics in Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
- Horizon Adaptive Offline Policy Learning via Value Stitching (NeurIPS 2026 · similarity 0.88)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.87)
- Efficient Off-Policy Learning for High-Dimensional Action Spaces (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.