Research map / Multi-agent systems and games / LLM agents

Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Robust and efficient agent execution · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Skill-Pro, which lets frozen LLM agents autonomously learn reusable procedural Skills (activation/execution/termination) from experience, evolving the skill pool via non-parametric PPO with semantic gradients, a PPO trust-region gate, and score-based pruning.

Problem

LLM agents re-derive solutions from scratch in recurring situations, and existing memory methods store episodic narratives that are costly and not directly executable or reusable; parametric RL risks forgetting.

Approach

Formalizes a Skill-MDP over a dynamic pool of natural-language Skills (activation, execution, termination conditions) and performs PPO-style optimization without weight updates: LLM-generated semantic gradients propose candidate Skills, a clipped-surrogate PPO Gate verifies them on historical trajectories, and online advantage-based scores prune the pool.

Key results

• Skill-Pro reaches 0.925 in-domain reuse rate on Mastermind-v0 versus 0.349 for the best RAG baseline, while storing only 816 total tokens.
• On ALFWorld training tasks, Skill-Pro achieves 0.900 success rate versus 0.700 for AWM and 0.680 for Expel.
• On BFCL v4, Skill-Pro reaches 0.433 accuracy versus 0.383 for ReAct and 0.367 for CoT.

Surprising findings

• Removing the PPO Gate lets all candidates through (100% pass rate) yet degrades reuse and performance, showing unconditional admission destabilizes training.
• Replacing score-based pruning with FIFO produces a negative online score, indicating the pool collapses without score-based maintenance.

Limitations

• Skills remain explicit natural-language procedures; implicit procedural execution is left to future work.
• Skill selection uses simple similarity or value heuristics and is kept fixed rather than learned.

Experimental setup

Datasets: ALFWorld TextArena Mastermind-v0

Benchmarks: ALFWorld Mastermind-v0 Mastermind-v0-Hard Mastermind-v0-Extreme BFCL v4

Metrics: success rate average return experience reuse rate total stored tokens retrieval ratio

Baselines: State ReAct CoT RAG Expel A-MEM AWM G-Memory

Builds on: PPO TextGrad Claude Agent Skills ReAct Chain-of-thought

Closest papers by meaning

• SkillOS: Learning Skill Curation for Self-Evolving Agents (NeurIPS 2026 · similarity 0.90)
• SkillMaster: Toward Autonomous Skill Mastery in LLM Agents (NeurIPS 2026 · similarity 0.90)
• From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills (NeurIPS 2026 · similarity 0.90)
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (NeurIPS 2026 · similarity 0.89)
• MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents (NeurIPS 2026 · similarity 0.89)
• Experience-Evolving Multi-Turn Tool-Use Agent with Hybrid Episodic–Procedural Memory (ICML 2026 · similarity 0.89)