Research map / Reinforcement learning / Offline RL
Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning (Abstract Reprint)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Action Impact Regularity (AIR), an MDP class where actions barely affect exogenous dynamics, and an FQI-based offline RL algorithm (FQI-AIR) with suboptimality bounds that avoid concentration coefficients.
Problem
Offline RL is hard for general MDPs due to poor data coverage and distribution shift; existing guarantees need strong coverage (concentration coefficients) or assumptions about data covering an optimal policy.
Approach
Assume actions have only limited (epsilon) impact on the exogenous state component while the endogenous component and reward are known or approximately modeled; then simulate counterfactual endogenous transitions on each observed exogenous state within FQI, giving bounds with no concentration-coefficient assumption.
Key results
- FQI-AIR significantly outperforms FQI, MBS-QI, CQL and IQL across three data collection policies in both simulated environments when epsilon_air=0.
- With epsilon_air=0.8 FQI-AIR still consistently outperforms baselines in most settings, except inventory with the learned policy.
- On real-world Prius and Bitcoin experiments FQI-AIR performs significantly better than CQL, IQL and FQI; FQI-AIR with a learned endogenous model remains effective.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FQI-AIR instead of FQI, MBS-QI, CQL, or IQL | Offline policy optimization · Optimal order execution and inventory management | policy performance | Helped |
| Use FQI-AIR instead of CQL, IQL, or FQI | Offline policy optimization · Bitcoin and Prius | policy performance | Helped |
| Use FQI-AIR instead of other offline methods | Offline policy optimization · Optimal order execution and inventory management, ε_air = 0.8 | policy performance | Mixed |
| Use FQI-AIR instead of trajectory-simulation API or actor-critic | Offline policy optimization · Optimal order execution and inventory management | policy performance | Helped |
| Replace the true endogenous model with a learned endogenous model (ablation) | FQI-AIR · Prius | policy performance | Hurt |
| Use a learned endogenous model with FQI-AIR rather than a learned full model (ablation) | Prius model-learning comparison · Prius | policy performance | Helped |
| Use FQI-AIR with the true endogenous model rather than model-based planning with the true endogenous model | Offline policy optimization · Prius, small data size | policy performance | Helped |
| Learn the endogenous model from one driving cycle rather than using the true model (ablation) | FQI-AIR · Prius | policy performance | No clear effect |
| Increase ε_air | FQI-AIR and offline baselines · Optimal order execution and inventory management | policy performance | Hurt |
| Increase the number of trajectories N | FQI-AIR policy evaluation · Simulated environments | policy evaluation error | Helped |
| Increase ε_air | FQI-AIR policy evaluation · Simulated environments | policy evaluation error | Hurt |
Surprising findings
- CQL and IQL fail when the data collection policy is far from optimal (constant policy), even though they are strong baselines elsewhere.
- A uniform random policy in the order execution environment achieves about 5, and MBS-QI's advantage over FQI arose from tie-breaking with uniform random actions.
Limitations
- Requires known or approximate endogenous model and known exogenous/endogenous separation; assumes AIR holds for all exogenous state-action pairs.
- Theoretical bound scales with sqrt(|S_end||A|) and worst-case epsilon_air loss, and is loose relative to experiments.
Experimental setup
Datasets: simulated optimal order execution (ARMA stock prices) simulated inventory management bitcoin historical prices (Kaggle G-Research crypto) Prius hybrid car driving cycles
Benchmarks: optimal order execution (simulated) inventory management (simulated) Bitcoin order execution Prius hybrid car control
Metrics: average return J(pi, M) policy evaluation error |J_hat - J|
Baselines: FQI MBS-QI CQL IQL model-based (MB) offline RL online API and actor-critic with trajectory simulation
Builds on: Fitted Q Iteration Exogenous MDPs MBS-QI CQL IQL
Closest papers by meaning
- Policy Regularization with Dataset Constraint for Offline Reinforcement Learning (ICML 2023 · similarity 0.90)
- Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning (ICML 2024 · similarity 0.89)
- Iteratively Refined Behavior Regularization for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- ContraDiff: Planning Towards High Return States via Contrastive Learning (ICLR 2025 · similarity 0.89)
- Offline-Online Reinforcement Learning for Linear Mixture MDPs (NeurIPS 2026 · similarity 0.88)
- Q-value Regularized Transformer for Offline Reinforcement Learning (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.