Research map / Reinforcement learning / MDPs and planning
Learning and Planning for Time-Varying MDPs Using Maximum Likelihood Estimation
Melkior Ornik, Ufuk Topcu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CCMLE instead of classical estimation (slow change, exp a) | transition probability estimator · Gridworld patrolling exp (a) | mean error: 0.11 -> 0.03 | Improved |
| use CCMLE instead of classical estimation when change is faster than the assumed bound (0.02 per step) | transition probability estimator · Gridworld patrolling exp (c) | mean error: 0.12 -> 0.06 | Improved |
| use CCMLE instead of classical estimation for a sudden switch | transition probability estimator · Gridworld patrolling exp (d) | mean error: 0.16 -> 0.1 | Improved |
| add forgetfulness to CCMLE (ablation) | CCMLE · Two-state periodic TVMDP | mean error: 0.06 -> 0.05 | Improved |
| add forgetfulness (sliding window) to classical estimation (ablation) | classical estimation · Two-state periodic TVMDP | mean error: 0.28 -> 0.14 | Improved |
| use CCMLE-based planning instead of classical estimation planning | patrolling agent · Gridworld patrolling planning, t=200 | walls visited: 18 -> 38 | Improved |
| add uncertainty-based learning bonus (beta>0) to CCMLE planning (ablation) | CCMLE planner · Two-state MDP planning | asymptotic average reward: 1 -> 1.15 | Improved |
| increase beta to a very high value (ablation) | CCMLE planner · Two-state MDP planning | asymptotic average reward: 1.15 -> 0.88 | Hurt |
| use forgetful CCMLE with uncertainty bonus instead of classical estimation in 5-arm bandit | bandit agent · Time-varying 5-armed bandit, 10000 steps | average collected reward: 25% higher | Improved |
| use CCMLE instead of classical estimation with misspecified bound epsilon | transition probability estimator · Wind flow estimation | average of maximal error: 0.4 -> 0.21 | Improved |
Closest papers by meaning
- Model-Based Exploration in Monitored Markov Decision Processes (ICML 2025 · similarity 0.87)
- Offline-Online Reinforcement Learning for Linear Mixture MDPs (NeurIPS 2026 · similarity 0.87)
- Online MDP with Prototypes Information: A Robust Adaptive Approach (AAAI 2025 · similarity 0.86)
- RL in Latent MDPs is Tractable: Online Guarantees via Off-Policy Evaluation (NeurIPS 2024 · similarity 0.86)
- Structure Matters: Dynamic Policy Gradient (NeurIPS 2025 · similarity 0.86)
- Planning and Learning in Average Risk-aware MDPs (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
