Research map / Continual learning / Online and replay-based
Skill Expansion and Composition in Parameter Space
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PSEC, a framework that stores skills as plug-and-play LoRA modules on a frozen diffusion policy and composes them in parameter space with a context-aware network, evaluated on safe offline RL (DSRL), continual policy shift (DMC), and dynamics shift (D4RL).
Problem
Autonomous agents need to reuse prior skills to learn new tasks efficiently and expand their skill sets, but prior methods either train skills from scratch, suffer catastrophic forgetting, or compose skills only at action or noise level with fixed weights.
Approach
Encode each skill as a LoRA module on a shared frozen diffusion policy, then merge LoRA modules in parameter space with state-dependent (context-aware) coefficients learned by a small composition network, instead of composing outputs at the noise or action level with fixed weights.
Key results
- On DSRL MetaDrive and BulletGym averages, PSEC reaches reward 0.50 and cost 0.03 on MetaDrive and 0.41 reward with 0.33 cost on BulletGym, the only method with safe cost while achieving top reward among safe agents in the reported table.
- PSEC outperforms noise-level (NSEC) and action-level (ASEC) composition in the DSRL and continual policy shift experiments; context-aware composition beats fixed composition across composition levels.
- In continual policy shift, LoRA rank 8 gave the best results; PSEC reaches good performance with under 50k gradient steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use parameter-level skill composition instead of action-level composition | PSEC · DSRL MetaDrive average | cost: 0.03 vs 2.02 | Helped |
| Use parameter-level skill composition instead of noise-level composition | PSEC · DSRL BulletGym average | cost: 0.33 vs 4.24 | Helped |
| Use parameter-level composition rather than noise-level composition | PSEC · DSRL BulletGym average | reward: 0.41 vs 0.62 | Hurt |
| Use context-aware composition rather than fixed composition (ablation) | PSEC · DMC Walker continual policy shift | performance | Helped |
| Add LoRA modules instead of replacing them with MLP modules (ablation) | PSEC · DMC Walker continual policy shift | performance | Helped |
| Use LoRA rank n=8 (ablation) | PSEC · DMC Walker continual policy shift | performance | Helped |
| Add the learned walking skill to the skill library before learning to run (S+W→R vs S→R) | PSEC · DMC Walker run | performance | Helped |
Surprising findings
- Parameter-level LoRA composition with the MLP variant still outperforms noise- and action-level composition, suggesting the composition level matters more than the module type.
Limitations
- Assumes the pretrained policy pi_0 covers a diverse distribution; otherwise LoRA adaptation may need more parameters.
- Maintaining an ever-growing skill library incurs computational and management costs; redundant skills are not pruned.
- Introduces extra LoRA hyperparameters (rank, scaling) and uses a simple context-aware composition network.
Experimental setup
Datasets: DSRL D4RL DeepMind Control Suite (walker stand/walk/run expert data)
Benchmarks: DSRL DeepMind Control Suite D4RL
Metrics: normalized reward normalized cost sample efficiency training efficiency
Baselines: FISOR CDT COptiDICE CPQ BC NSEC ASEC Scratch CQL IQL MOPO DOGE
Builds on: LoRA Diffusion Policy IDQL energy-based compositional diffusion
Closest papers by meaning
- SkillOS: Learning Skill Curation for Self-Evolving Agents (NeurIPS 2026 · similarity 0.87)
- Efficient Skill Grounding via Code Refactoring with Small Language Models (ICML 2026 · similarity 0.87)
- SkillMaster: Toward Autonomous Skill Mastery in LLM Agents (NeurIPS 2026 · similarity 0.87)
- Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents (ICML 2026 · similarity 0.87)
- Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning (ICLR 2026 · similarity 0.86)
- Policy Compatible Skill Incremental Learning via Lazy Learning Interface (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.