Research map / Reinforcement learning / Policy gradient and actor-critic
Oracle-Efficient Reinforcement Learning for Max Value Ensembles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives an oracle-efficient algorithm (MaxIteration) that learns a policy competitive with approximate max-following over a set of constituent policies, using only a squared-error value regression oracle, and validates it on CompoSuite robotic manipulation tasks.
Problem
Improving upon a set of heuristic constituent policies in large or infinite state spaces with provable efficiency, without assuming access to optimal policy or max-following value functions.
Approach
Incrementally builds an improved policy step by step: at each step h, samples states by executing the current policy for h steps, queries a regression oracle for each constituent's value function on that distribution, and follows the constituent with highest estimated value at step h. This competes with approximate max-following policies using only value-function regression on samplable distributi
Key results
- On selected CompoSuite tasks, MaxIteration yields returns at least as good as the best constituent policy in all cases, per the paper.
- In some tasks max-following solves the task successfully even when no constituent policy achieves success.
- MaxIteration consistently gives greater return improvement than IQL fine-tuning given the same number of episodes; MAPS baseline failed to obtain non-trivial return despite tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MaxIteration max-following instead of fine-tuning IQL | CompoSuite tasks · CompoSuite | cumulative return | Helped |
| Use MaxIteration max-following instead of the best constituent policy | MaxIteration · CompoSuite tasks | cumulative return | Helped |
| Use max-following to combine constituent policies | MaxIteration · Selected CompoSuite tasks | success rate | Mixed |
| Select among checkpointed constituent policies with MaxIteration | MaxIteration · DM Control benchmarks | cumulative return | No clear effect |
Surprising findings
- Increased return from max-following does not always translate into task success.
- The MAPS baseline failed to obtain non-trivial return on CompoSuite despite tuning.
Limitations
- Assumes efficient batch learnability of constituent value functions, which may not be feasible in practice.
- Max-value ensembling is sensitive to small value differences between constituents; softmax alternatives not studied.
- Results limited to fixed-horizon episodic MDPs; partially observable and discounted infinite-horizon settings not covered.
Experimental setup
Datasets: CompoSuite offline expert datasets
Benchmarks: CompoSuite DM Control
Metrics: cumulative return success rate
Baselines: Implicit Q-learning fine-tuning MAPS
Builds on: Implicit Q-learning MAPS MAMBA
Closest papers by meaning
- Maximum Likelihood Reinforcement Learning (ICML 2026 · similarity 0.90)
- Breaking the Computational Barrier: Provably Efficient Actor–Critic for Low-Rank MDPs (ICML 2026 · similarity 0.90)
- To the Max: Reinventing Reward in Reinforcement Learning (ICML 2024 · similarity 0.90)
- Nearly Minimax Optimal Reinforcement Learning for Linear Markov Decision Processes (ICML 2023 · similarity 0.89)
- On Oracle-Efficient PAC RL with Rich Observations (NeurIPS 2018 · similarity 0.89)
- Blending Imitation and Reinforcement Learning for Robust Policy Improvement (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.