Research map / Multi-agent systems and games / LLM agents
From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematic utility-grounded study of model-generated domain-level agent skills across experience generation, extraction, and consumption over five domains, finding mixed transfer and proposing a validated-rubric meta-skill that improves extraction.
Problem
Model-generated agent skills are proliferating, but no study examines the full skill lifecycle (experience generation, extraction, consumption) to determine whether such skills actually help, when, and why.
Approach
Measure skill utility as the downstream performance delta from injecting an extractor's skill into a target, disentangling extractor effects (Extraction Efficacy) and target effects (Target Evolvability); then derive a validated three-dimension rubric from high-gap skill pairs and use it as a meta-skill in the extractor prompt.
Key results
- Model-generated skills improve performance in 75% of extractor-target entries, with 25% showing negative transfer; ALFWorld is most fragile (47% negative).
- Unguided GPT-5.4 judge selects the higher-utility skill only 46.4% of the time; 15.8% on pairs with gap >=5pp.
- Validated three-dimension rubric raises judge accuracy to 73.8%, and the meta-skill improves all nine cells by +1.55pp on average, versus -0.59pp for the plausibility rubric.
Surprising findings
- Textual plausibility is anti-correlated with downstream utility: on pairs with gap >=5pp the LLM judge picks the better skill only 15.8% of the time.
- A stronger baseline model is not a better extractor; GPT-5.4 ranks last in extraction efficacy on SpreadsheetBench while Gemini-3.1-Flash-Lite ranks first.
Limitations
- Single consolidated domain-level skill per setting; does not study skill libraries, retrieval, or multi-skill selection.
- Minimal harness with skills injected via system prompt, so results may not transfer to richer agent scaffolds.
- Many results rely on closed API models with unreleased internals and one judge model (GPT-5.4).
Experimental setup
Datasets: ALFWorld SpreadsheetBench SWE-bench-Verified SEAL-0 BFCL-v4
Benchmarks: ALFWorld SpreadsheetBench SWE-bench-Verified SEAL-0 BFCL-v4
Metrics: performance delta (pp) Extraction Efficacy Target Evolvability pairwise selection accuracy better-rate
Baselines: plausibility rubric guided extraction unguided extractor prompt no-skill baseline
Builds on: Trace2Skill SkillCraft Agent Skills open standard
Closest papers by meaning
- SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (NeurIPS 2026 · similarity 0.91)
- SkillOpt: Executive Strategy for Self-Evolving Agent Skills (NeurIPS 2026 · similarity 0.90)
- Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents (ICML 2026 · similarity 0.90)
- SkillMaster: Toward Autonomous Skill Mastery in LLM Agents (NeurIPS 2026 · similarity 0.89)
- SkillOS: Learning Skill Curation for Self-Evolving Agents (NeurIPS 2026 · similarity 0.88)
- SKILL-MIX: a Flexible and Expandable Family of Evaluations for AI Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.