Research map / Reinforcement learning / Policy gradient and actor-critic
Challenging Common Assumptions in Convex Reinforcement Learning
Mirco Mutti, Riccardo De Santi, Piersilvio De Bartolomeis, Marcello Restelli
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize the finite-trials objective over empirical state distributions instead of the infinite-trials objective over their expectation | Convex RL · Pure exploration | state entropy | Improved |
| Optimize the finite-trials objective over empirical state distributions instead of the infinite-trials objective over their expectation | Convex RL · Risk-averse RL | CVaR | Improved |
| Optimize the finite-trials objective over empirical state distributions instead of the infinite-trials objective over their expectation | Convex RL · Imitation learning | KL divergence to expert distribution | Improved |
| Use the finite-trials objective rather than the infinite-trials objective | Convex RL · Finite-trials convex RL | approximation error: 4 L T \sqrt{ \frac{2 S \log ( 4T / \delta)}{n}} | Mixed |
| Use history-dependent policies instead of restricting to stationary policies | Single-trial convex RL · Pure exploration | state entropy | Improved |
| Use history-dependent policies instead of restricting to stationary policies | Single-trial convex RL · Risk-averse RL | CVaR | Improved |
| Use history-dependent policies instead of restricting to stationary policies | Single-trial convex RL · Imitation learning | KL divergence to expert distribution | Improved |
| Replace the state with the full trajectory history in an extended MDP | Single-trial convex RL · Finite-horizon convex MDP | policy-space size: |S_ℓ| > S^T | Hurt |
| Replace the nonlinear convex objective with a linear reward objective | Convex RL · Linear RL, any number of trials | optimal-policy equivalence | No clear effect |
Closest papers by meaning
- Convex Reinforcement Learning in Finite Trials (JMLR 2023 · similarity 0.96)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.89)
- Global Reinforcement Learning : Beyond Linear and Convex Rewards via Submodular Semi-gradient Methods (ICML 2024 · similarity 0.88)
- Hybrid Reinforcement Learning Breaks Sample Size Barriers In Linear MDPs (NeurIPS 2024 · similarity 0.88)
- The Effective Horizon Explains Deep RL Performance in Stochastic Environments (ICLR 2024 · similarity 0.88)
- Beyond Optimism: Exploration With Partially Observable Rewards (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
