Research map / Reinforcement learning / Offline RL
Variational Consequence-Driven Offline Reinforcement Learning
Wen Jiang, Ke Jiang, Jin Wang, Xiaoyang Tan
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a variational, consequence-driven offline RL approach that constrains policies by the consequences of actions (dynamic consistency) rather than only by action-distribution divergence.
Problem
Offline RL typically handles distribution shift with divergence constraints on action distributions, which fails to ensure dynamic consistency: small action deviations can cause very different transitions that drive the agent away from the offline dataset.
Approach
Shifts the offline RL constraint from action-distribution divergence to the consequences of actions, using a variational formulation to enforce dynamic consistency with the offline data. Abstract is truncated, so details beyond this are unknown.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| constrain induced state-consequence distribution via latent divergence instead of action-distribution matching | TD3-based offline RL · D4RL MuJoCo total | normalized score: 773.0 (SPOT) -> 811.0 (StBR) | Improved |
| use StBR latent consequence constraint | TD3 · D4RL AntMaze total | normalized score: 501.0 (FQL) -> 534.2 (StBR) | Improved |
| use StBR latent consequence constraint | TD3 · D4RL Adroit human total | normalized score: 83.6 (IQL) -> 110.5 (StBR) | Improved |
| explicit next-state prediction instead of implicit latent consequence constraint | ENSP baseline · Adroit (45-dim) | normalized score: ENSP -3.0 vs StBR 110.5 total | Hurt |
| remove the bidirectional action IB constraint (degenerates to standard VIB) (ablation) | StBR · MuJoCo, AntMaze, Adroit | percent performance difference | Improved |
| use a one-step generative stochastic policy (FQL-based) instead of deterministic policy (ablation) | StBR · AntMaze (sparse reward) | performance | Improved |
| use a one-step generative stochastic policy instead of deterministic policy (ablation) | StBR · near-deterministic optimal-action tasks | performance | Hurt |
| increase beta (information compression) for high-dimensional tasks (ablation) | StBR · hopper vs antmaze and pen | normalized score (optimal beta): optimal beta 10^-4 (hopper) -> 10^-2 (antmaze, pen) | Mixed |
| increase environment action noise variance (ablation) | StBR on hopper-medium · hopper-medium with injected noise | optimal beta | Mixed |
| increase lambda together with beta (ablation) | StBR · lambda-beta sweep | optimal beta | Mixed |
| vary latent dimension from 0.5x to 2x state dimension (ablation) | StBR · latent dimension sensitivity | performance | Mixed |
| sweep lambda from 0.015 to 0.5 (ablation) | StBR vs baselines · hopper-medium-v2, antmaze-large-diverse-v2 | normalized score | Improved |
Closest papers by meaning
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- Behavior-Regularized Diffusion Policy Optimization for Offline Reinforcement Learning (ICML 2025 · similarity 0.91)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.90)
- Robust Offline Reinforcement Learning with Linearly Structured $f$-Divergence Regularization (ICML 2025 · similarity 0.90)
- Constrained Latent Action Policies for Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learning (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
