Research map / Reinforcement learning / Offline RL
Learning Versatile Skills with Curriculum Masking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CurrMask, a curriculum-based masked prediction pretraining for offline RL that uses block-wise masking with mixed mask ratios and block sizes, scheduled by an EXP3 bandit driven by target loss decrease.
Problem
Masked prediction pretraining for offline RL/sequential decision making lacks a principled way to choose and order masking schemes of varying complexity, and random token masking fails to capture long-term dependencies in state-action trajectories.
Approach
Combine block-wise masking over a pool of 100 (mask ratio, block size) schemes and adaptively schedule them with an EXP3 multi-armed bandit whose reward is the rescaled decrease in validation reconstruction loss, creating an automatic masking curriculum.
Key results
- Skill prompting: CurrMask averages 75.0 reward vs 56.7 for MaskDP and 68.2 for the best baseline Mixed-prog, best on 7 of 9 tasks.
- Goal-conditioned planning: CurrMask averages 8.60 L2 distance vs 8.71 for MaskDP, best on 8 of 9 tasks (small margin).
- Offline RL: CurrMask generally outperforms masked pretraining baselines and GPT-style baselines, with a reported 4.7% wall-clock overhead for 100k gradient steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use curriculum masking instead of the strongest baseline | CurrMask · Skill prompting (Average) | reward: 68.2 -> 75.0 | Helped |
| Use curriculum masking instead of the strongest baseline | CurrMask · Goal-conditioned planning (Average) | distance: 8.71 -> 8.60 | Helped |
| Add block-wise masking rather than token-wise masking (ablation) | masked prediction · Skill prompting (Average) | reward: 56.7 -> 35.0; 56.7 -> 73.4 | Mixed |
| Increase fixed block size from 5 to 20 (ablation) | block-wise masked prediction · Skill prompting (Average) | reward: 35.0 -> 73.4 | Helped |
| Use a manually progressive block-size curriculum instead of uniform mixed masking (ablation) | masked prediction · Goal-conditioned planning and offline RL | performance | Hurt |
| Adaptively schedule masking schemes using curriculum masking instead of uniform mixed masking | masked prediction · Skill prompting and goal-conditioned planning | performance | Helped |
Surprising findings
- Manually designed curricula (Mixed-prog) do not consistently beat uniform random mixing, and the reversed curriculum Mixed-inv performs poorly on skill prompting.
- Mixed block-size schemes and Mixed-prog are worse than MaskDP on goal-conditioned planning, although they help on skill prompting.
Limitations
- Computational overhead from the bandit scheduler (4.7% wall-clock for 100k steps).
- Gains may depend on the structure of the environment; evaluated only on MuJoCo/DMC state-based tasks, not image-based RL.
- Goal-conditioned planning gains over MaskDP are marginal.
Experimental setup
Datasets: DeepMind control suite walker multi-task replay data (12M steps) quadruped multi-task replay data jaco multi-task replay data ProtoRL replay buffer offline RL datasets
Benchmarks: DeepMind control suite walker DeepMind control suite quadruped DeepMind control suite jaco
Metrics: episodic reward (skill prompting) L2 distance to goal (goal-conditioned planning) offline RL return
Baselines: MaskDP MTM Mixed Mixed-prog Mixed-inv GPT Goal-GPT Random token masking
Builds on: MaskDP MTM TD3 EXP3 ProtoRL
Closest papers by meaning
- Robust Policy Learning via Offline Skill Diffusion (AAAI 2024 · similarity 0.85)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.85)
- Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization (NeurIPS 2026 · similarity 0.85)
- RLP: Reinforcement as a Pretraining Objective (ICLR 2026 · similarity 0.85)
- How does RL Post-training Induce Skill Composition? A Case Study on Countdown (NeurIPS 2026 · similarity 0.85)
- Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learning (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.