Research map / Reinforcement learning / Offline RL
ROIDICE: Offline Return on Investment Maximization for Efficient Decision Making
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ROIDICE, an offline policy optimization method that maximizes the return-on-investment (return divided by accumulated cost) of a policy by casting ROI maximization as linear-fractional programming, converting it to an LP via Charnes-Cooper, and solving it with a DICE-style convex regularizer.
Problem
Maximizing the efficiency of a policy, measured as the ratio of return to accumulated cost, in an MDP with a cost function using only a fixed offline dataset. The ratio of two long-term expected values makes gradient-based policy optimization hard.
Approach
Reformulates ROI maximization as linear-fractional programming and applies the Charnes-Cooper transformation to obtain ROI-LP over stationary distributions. It then adds a new convex f-divergence-like regularizer f(x,t) that measures distribution shift while preserving convexity, yielding a DICE-style offline algorithm with weighted behavior cloning for policy extraction.
Key results
- On D4RL Hopper medium, ROIDICE reaches ROI 9.21±0.49 versus 3.58±0.03 for OptiDICE and 7.37±0.19 for COptiDICE (50th percentile cost).
- In random tabular MDPs, ROIDICE is compared against OptiDICE (offline RL) and COptiDICE (offline constrained RL) over dataset sizes 10 to 2000 trajectories; OptiDICE attains higher return but at higher cost.
- On SafetyGym CarGoal, ROIDICE achieves ROI 0.160±0.02 versus 0.119±0.01 for COptiDICE (50th percentile).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ROIDICE instead of the strongest reported baseline | offline policy optimization · Hopper medium (H-m) | ROI: 7.37±0.19 -> 9.21±0.49 | Helped |
| Use ROIDICE instead of the strongest reported baseline | offline policy optimization · Hopper medium-expert (H-m-e) | ROI: 7.88±0.21 -> 8.29±0.18 | Helped |
| Use ROIDICE instead of the strongest reported baseline | offline policy optimization · Hopper expert (H-e) | ROI: 7.99±0.18 -> 8.51±0.25 | Helped |
| Use ROIDICE instead of the strongest reported baseline | offline policy optimization · Walker2D medium (W-m) | ROI: 2.92±0.48 -> 3.06±0.14 | Helped |
| Use ROIDICE instead of the strongest reported baseline | offline policy optimization · Walker2D medium-expert (W-m-e) | ROI: 4.03±0.06 -> 4.21±0.78 | Helped |
| Use ROIDICE instead of the strongest reported baseline | offline policy optimization · Walker2D expert (W-e) | ROI: 4.59±0.02 -> 5.01±0.42 | Helped |
| Use ROIDICE instead of the strongest reported baseline | offline policy optimization · Halfcheetah medium (HC-m) | ROI: 8.04±0.4 -> 8.48±0.19 | Helped |
| Use ROIDICE instead of the strongest reported baseline | offline policy optimization · Halfcheetah medium-expert (HC-m-e) | ROI: 9.04±1.03 -> 10.17±0.4 | Helped |
| Use ROIDICE instead of the strongest reported baseline | offline policy optimization · Halfcheetah expert (HC-e) | ROI: 12.74±0.13 -> 12.69±0.09 | Hurt |
| Use ROIDICE instead of the strongest reported baseline | offline policy optimization · FinRL medium (F-M) | ROI: 24.4±8.5 -> 43.2±13.13 | Helped |
| Use ROIDICE instead of the strongest reported baseline | offline policy optimization · FinRL high (F-H) | ROI: 46.5±8.97 -> 47.2±8.67 | Helped |
| Use ROIDICE instead of COptiDICE 50th | offline constrained policy optimization · CarGoal | ROI: 0.119±0.01 -> 0.160±0.02 | Helped |
Surprising findings
- OptiDICE, which maximizes return alone, achieves the highest return in several tasks yet has substantially lower ROI than ROIDICE because it ignores cost.
- Offline constrained RL methods (CDT, CPQ, VOCE) often show low or negative ROI, which the authors attribute to cost overestimation.
Limitations
- Performance depends on offline dataset quality and coverage, and the method is sensitive to reward and cost function design.
- With a constant cost the method reduces to standard return maximization; with highly varying costs it can exhibit overly cost-averse behavior.
- The regularization strength alpha must be tuned per task and dataset, and performance varies strongly with alpha in some domains.
Experimental setup
Datasets: D4RL Hopper D4RL Walker2D D4RL Halfcheetah NeoRL FinRL OpenAI SafetyGym CarGoal and PointPush (PPO-Lagrangian data) Random tabular MDPs
Benchmarks: D4RL Hopper D4RL Walker2D D4RL Halfcheetah NeoRL FinRL OpenAI SafetyGym CarGoal OpenAI SafetyGym PointPush Random tabular MDP
Metrics: ROI return accumulated cost
Baselines: OptiDICE COptiDICE CDT VOCE CPQ Behavior cloning on top-ROI trajectories
Builds on: OptiDICE COptiDICE DICE-RL framework Charnes-Cooper transformation V-LP
Closest papers by meaning
- Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning (Abstract Reprint) (AAAI 2024 · similarity 0.85)
- AETDICE: Unified Framework and Offline Optimization for Nonlinear Multi-Objective RL (NeurIPS 2026 · similarity 0.85)
- ContraDiff: Planning Towards High Return States via Contrastive Learning (ICLR 2025 · similarity 0.85)
- Warm-up Free Policy Optimization: Improved Regret in Linear Markov Decision Processes (NeurIPS 2024 · similarity 0.84)
- Policy Learning for Balancing Short-Term and Long-Term Rewards (ICML 2024 · similarity 0.84)
- Achieving Tractable Minimax Optimal Regret in Average Reward MDPs (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.