Research map / Large language models / Agents, code and math
Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Skill Set Optimization (SSO), which constructs and prunes transferable skills from high-reward subtrajectories and injects them in-context to improve GPT-4 actors on ScienceWorld and a custom NetHack task.
Problem
LLM actors in interactive sequential decision-making environments cannot easily use environment reward signals for continual improvement, since finetuning is costly or impossible with black-box models and in-context examples scale poorly.
Approach
Extract similar high-reward subtrajectories across past episodes, score and sample non-overlapping sets via beam search, generate abstract subgoals and instructions as skills, and prune skills whose observed discounted future return drops below a threshold.
Key results
- SSO outperforms baselines on the custom NetHack task by 40%.
- SSO outperforms the previous state-of-the-art CLIN on ScienceWorld by an average of 35%.
- Skill refinement and reward-weighted scoring each reduce performance when ablated but still outperform the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide learned skills in context | SSO actor · ScienceWorld: Fill container | action accuracy: 0.07 -> 0.37 | Helped |
| Provide learned skills in context | SSO actor · ScienceWorld: Heat substance | action accuracy: 0.04 -> 0.22 | Helped |
| Provide learned skills in context | SSO actor · ScienceWorld: Mix ingredients | action accuracy: 0.30 -> 0.36 | Helped |
| Add skill set refinement that prunes skills with low observed reward (ablation) | SSO · ScienceWorld: Melting Temperature | task performance | Helped |
| Use environment reward when scoring skills (ablation) | SSO · ScienceWorld: Melting Temperature | task performance | Helped |
| Extract pairs of similar subtrajectories (ablation) | SSO · ScienceWorld: Melting Temperature | task performance | Helped |
| Add skill set refinement that prunes skills with low observed reward (ablation) | SSO · Custom NetHack task | task performance | Helped |
| Use environment reward when scoring skills (ablation) | SSO · Custom NetHack task | task performance | Helped |
| Extract pairs of similar subtrajectories (ablation) | SSO · Custom NetHack task | task performance | Helped |
Surprising findings
- Skill self-reporting by the LLM actor is correct only about 70% of the time.
- Fewshot trajectory examples performed worse than skills despite the actor's context being about 5x longer for fewshot.
Limitations
- Embedding-based similarity may be less effective with distracting state information or low-level actions.
- Best performance requires intermediate subgoal rewards in the environment.
- No built-in use of negative feedback outside skill refinement.
Experimental setup
Datasets: ScienceWorld NetHack (custom MiniHack level)
Benchmarks: ScienceWorld NetHack custom MiniHack task
Metrics: task score (0-100) action accuracy skill self-report rate
Baselines: ReAct Reflexion CLIN fewshot trajectories
Builds on: ReAct Reflexion CLIN Voyager MiniHack
Closest papers by meaning
- Expanding LLM Agent Boundaries with Strategy-Guided Exploration (NeurIPS 2026 · similarity 0.88)
- Enhancing Decision-Making of Large Language Models via Actor-Critic (ICML 2025 · similarity 0.87)
- The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement (NeurIPS 2025 · similarity 0.87)
- SkillMaster: Toward Autonomous Skill Mastery in LLM Agents (NeurIPS 2026 · similarity 0.87)
- On the Modeling Capabilities of Large Language Models for Sequential Decision Making (ICLR 2025 · similarity 0.87)
- Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.