Research map / Large language models / Reasoning and chain of thought

Procedural Memory Distillation: Online Reflection for Self-Improving Language Models

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Procedural Memory Distillation (PMD), which builds three-level online procedural memory from a model's own rollouts, uses it to condition an on-policy self-teacher, and distills it into memory-free weights, improving over SDPO on SciKnowEval and LiveCodeBench.

Problem

Episode-local RLVR and self-distillation methods (GRPO, SDPO) discard cross-episode procedural information from repeated attempts at related problems, so recurring strategies and failure modes are not retained or reused during training.

Approach

Build experience, insight, and behavior memory online from the policy's own rollouts, co-evolve it with the policy, and use it only to condition the self-teacher so the memory-free student internalizes it via reverse-KL self-distillation.

Key results

• PMD improves SciKnowEval AVG over SDPO from 74.4 to 77.2 on Qwen3-8B (+3.8%) and from 69.5 to 73.3 on OLMo3-Instruct-7B (+5.5%).
• PMD improves LiveCodeBench v6 over SDPO from 47.9 to 51.7 on Qwen3-8B (+7.9%) and from 45.0 to 51.1 on OLMo3-Instruct-7B (+13.6%).
• Freezing memory or policy trails PMD by more than 10 points on SciKnowEval domains (Frozen Memory + Evolving Policy 65.0 vs 77.2 AVG on Qwen3-8B).

Surprising findings

• Evolving memory with a frozen policy barely improves over base (54.0 vs 47.9 AVG), indicating memory helps only when distilled into weights.
• On LiveCodeBench, transient (non-persistent) reflection gives almost no gain over SDPO (+0.2pp), with persistence accounting for nearly all improvement.

Limitations

• Evaluation limited to repeated training over fixed task distributions with two verifiable domains.
• Not validated on long-horizon agentic environments with heterogeneous tasks.
• Training step time increases by 1.45x over SDPO due to longer teacher prompts and memory extraction.

Experimental setup

Datasets: SciKnowEval LiveCodeBench

Benchmarks: SciKnowEval LiveCodeBench

Metrics: avg@16 score@4 maj@k best@k

Baselines: GRPO SDPO PMD-Transient Evolving Memory + Frozen Policy Frozen Memory + Evolving Policy

Builds on: SDPO GRPO on-policy distillation OPCD OEL

Closest papers by meaning

• Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents (NeurIPS 2026 · similarity 0.89)
• Learning from Language Feedback via Variational Policy Distillation (NeurIPS 2026 · similarity 0.88)
• MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents (NeurIPS 2026 · similarity 0.88)
• Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation (NeurIPS 2026 · similarity 0.88)
• From Player to Master: Enhancing Test-Time Learning of LLM Agents via Reinforcement Learning over Memory (ICML 2026 · similarity 0.88)
• EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.88)