Research map / Large language models / Agents, code and math
MaestroMotif: Skill Design from Artificial Intelligence Feedback
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
MaestroMotif uses LLM AI feedback to design skill rewards from natural-language descriptions, trains skills with RL, and uses LLM-generated code policies to compose skills zero-shot on NetHack tasks.
Problem
Designing low-level skills for LLM-controlled agents requires heavy human labor for reward engineering and skill construction; the paper asks how to turn natural-language skill descriptions into usable skills and compose them zero-shot for complex language-specified tasks in NetHack.
Approach
Combine Motif-style LLM preference feedback (per skill, via skill-specific modifiers) to derive skill reward functions with LLM-written code for skill initiation/termination and a training-time and deployment-time policy over skills, so that new tasks are solved by recomposing skills in code without further training.
Key results
- On navigation tasks MaestroMotif reaches 46% success on Gnomish Mines, 29% on Delphi, and 7.2% on Minetown, versus at most 9% for the best baseline (Motif) and near 0% for zero-shot LLM policy.
- On interaction tasks MaestroMotif collects 0.66 transactions, 0.47 price identifications, and 1.60 BUC identifications, far above Motif (0.08, 0.02, 0.05).
- Results are averaged over nine seeds; performance of the policy over skills improves with the scale of the LLM code generator.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MaestroMotif's zero-shot skill recomposition instead of task-specific Motif training | NetHack agent · Gnomish Mines | success rate: 9% ± 2.30% -> 46% ± 1.70% | Helped |
| Use MaestroMotif's zero-shot skill recomposition instead of task-specific Motif training | NetHack agent · Delphi | success rate: 2% ± 0.70% -> 29% ± 1.20% | Helped |
| Use MaestroMotif's zero-shot skill recomposition instead of task-specific Motif training | NetHack agent · Minetown | success rate: 0% ± 0.00% -> 7.2% ± 0.50% | Helped |
| Use MaestroMotif's zero-shot skill recomposition instead of task-specific Motif training | NetHack agent · Transactions | collected objects: 0.08 ± 0.00 -> 0.66 ± 0.01 | Helped |
| Use MaestroMotif's zero-shot skill recomposition instead of task-specific Motif training | NetHack agent · Price Identified | collected objects: 0.02 ± 0.00 -> 0.47 ± 0.01 | Helped |
| Use MaestroMotif's zero-shot skill recomposition instead of task-specific Motif training | NetHack agent · BUC Identified | collected objects: 0.05 ± 0.00 -> 1.60 ± 0.01 | Helped |
| Replace one-hot skill conditioning with separate policy heads (ablation) | MaestroMotif skill network · NetHack task categories | normalized performance | Hurt |
| Train each skill in a separate episode instead of learning all skills simultaneously (ablation) | MaestroMotif · NetHack task categories | performance | Hurt |
| Scale up the LLM code generator (ablation) | MaestroMotif · NetHack tasks | performance | Helped |
Surprising findings
- Score-maximizing agents, including behavioral cloning from experts, rarely exhibit the interesting behaviors targeted by the benchmark despite collecting significant score.
- Using separate policy heads per skill caused a collapse in performance, whereas one-hot skill conditioning in a shared network worked well.
Limitations
- Agent behaviors are limited to the expressiveness of the predefined skill set; a composition of skills may not represent a satisfactory policy.
- Skill training depends on LLM-generated preferences and the design of skill descriptions by a human designer.
Experimental setup
Datasets: Motif preference dataset Dungeons and Data
Benchmarks: NetHack Learning Environment navigation tasks NetHack interaction tasks NetHack composite tasks
Metrics: success rate number of collected objects normalized performance
Baselines: LLM Policy (ReAct) Motif Embedding similarity reward RL with task reward + game score behavioral cloning score maximization
Builds on: Motif PPO ReAct Sample Factory Voyager
Closest papers by meaning
- SkillOpt: Executive Strategy for Self-Evolving Agent Skills (NeurIPS 2026 · similarity 0.87)
- Motif: Intrinsic Motivation from Artificial Intelligence Feedback (ICLR 2024 · similarity 0.87)
- Language Guided Skill Discovery (ICLR 2025 · similarity 0.87)
- SKILL-MIX: a Flexible and Expandable Family of Evaluations for AI Models (ICLR 2024 · similarity 0.87)
- From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills (NeurIPS 2026 · similarity 0.86)
- Skill Set Optimization: Reinforcing Language Model Behavior via Transferable Skills (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.