Research map / Large language models / Reasoning and chain of thought

Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Skill-SD, which summarizes an agent's own trajectories into natural-language skills that condition only a self-distillation teacher, combined with GRPO and an importance-weighted reverse-KL loss for stable multi-turn agent RL.

Problem

Sample-inefficient RL for multi-turn LLM agents under sparse, long-horizon rewards; naive on-policy self-distillation with a fixed privileged teacher does not capture diverse valid strategies and combining it with RL often collapses training.

Approach

Trajectory-derived natural-language skills condition only the teacher (student acts under plain prompt), a UCB-selected skill per task, an importance-weighted sampled-token reverse-KL (rho * k3) distillation loss, and dynamic synchronization of the teacher with the latest student checkpoint.

Key results

• Skill-SD reaches 64.9% accuracy on AppWorld and 62.5% on Sokoban, +14.0 and +10.9 points over vanilla GRPO.
• Skill-SD outperforms vanilla OPD by +42.1 (AppWorld) and +40.6 (Sokoban avg-level gains as reported in abstract) in accuracy.
• Off-policy (teacher-owned rollout) variants collapse mid-training (Sokoban accuracy 12.5% for frozen teacher), while on-policy dynamic teacher is best.

Surprising findings

• Skill-augmented GRPO (skills in student prompt) underperforms vanilla GRPO substantially, e.g. 20.3% vs 51.6% on Sokoban.
• Off-policy teacher-owned rollouts collapse to base-model-level performance on Sokoban despite strong early gains.

Limitations

• Skill retrieval uses a simple UCB bandit rather than learned or semantic retrieval.
• Uses sampled-token distillation rather than full-vocabulary distillation.
• Gradient analysis is token-level and per-update; trajectory-level bounds remain open.

Experimental setup

Datasets: AppWorld Sokoban (gym-sokoban generated levels)

Benchmarks: AppWorld dev set (57 tasks) Sokoban test set (64 levels)

Metrics: accuracy (pass@1) completion rate

Baselines: Vanilla GRPO Skill-Augmented GRPO Vanilla OPD Base model (Qwen3-4B-Instruct-2507)

Builds on: GRPO on-policy distillation DAPO k3 KL estimator UCB1

Closest papers by meaning

• SkillMaster: Toward Autonomous Skill Mastery in LLM Agents (NeurIPS 2026 · similarity 0.90)
• Privileged Information Distillation for Language Models (ICML 2026 · similarity 0.90)
• From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory (ICML 2026 · similarity 0.89)
• Reinforce LLM Reasoning through Multi-Agent Reflection (ICML 2025 · similarity 0.89)
• LLM-as-a-Tutor: Policy-Aware Prompt Adaptation for Non-Verifiable RL (NeurIPS 2026 · similarity 0.89)
• Self-evolving LLM agents with in-distribution Optimization (ICML 2026 · similarity 0.89)