Research map / Reinforcement learning / MDPs and planning

Decision-Focused Learning in MDPs: An Occupancy Measure Approach

NeurIPS 2026 · Reinforcement learning › MDPs and planning · approach: Average-reward and regularized MDPs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes decision-focused learning for MDPs by reformulating the MDP as an occupancy-measure LP whose feasible region is induced by predicted dynamics, enabling scalable differentiation through the LP instead of the Bellman KKT system.

Problem

Decision-focused learning for MDPs with predicted parameters is limited in scalability because existing methods differentiate through Bellman KKT conditions, requiring a linear system over all state-action pairs.

Approach

Reformulate the MDP as an occupancy-measure linear program whose feasible region depends on predicted dynamics, and differentiate the LP solution (via LP-layer Jacobians under non-degeneracy and fixed-basis assumptions) to train predictors on decision loss; variants include sketching, sensitivity analysis, and QP relaxations.

Limitations

• LP gradient is discontinuous at basis boundaries, which is hard to handle in practice
• Closed-form Jacobian derived under non-degeneracy and fixed-basis assumptions

Experimental setup

Datasets: Inventory Cliff Walking CartPole

Benchmarks: Inventory Cliff Walking CartPole

Metrics: wall-clock training time decision quality

Baselines: two-stage KKT-based DFL for MDPs

Builds on: decision-focused learning for MDPs via KKT differentiation qpth differentiable QP

Closest papers by meaning

• Learning MDPs from Features: Predict-Then-Optimize for Sequential Decision Making by Reinforcement Learning (NeurIPS 2021 · similarity 0.91)
• Achieving $\tilde{O}(1/\epsilon)$ Sample Complexity for Constrained Markov Decision Process (NeurIPS 2024 · similarity 0.87)
• Structure Matters: Dynamic Policy Gradient (NeurIPS 2025 · similarity 0.87)
• Solver-Free Decision-Focused Learning for Linear Optimization Problems (NeurIPS 2025 · similarity 0.87)
• Feasibility-Aware Decision-Focused Learning for Predicting Parameters in the Constraints (NeurIPS 2025 · similarity 0.87)
• Learning General Parameterized Policies for Infinite Horizon Average Reward Constrained MDPs via Primal-Dual Policy Gradient Algorithm (NeurIPS 2024 · similarity 0.87)