Research map / Large language models / Reasoning and chain of thought

Learning from Language Feedback via Variational Policy Distillation

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Variational Policy Distillation (VPD), a co-evolving variational EM framework where a feedback-conditioned shared-weight teacher is refined via unpaired preference optimization (BCO) and distilled into the student via token-level KL on on-policy rollouts.

Problem

Self-distillation methods that learn from language feedback (e.g., SDPO) use a fixed, zero-shot feedback-conditioned teacher, which plateaus as the student improves, and RLVR suffers from sparse outcome rewards and exploration bottlenecks.

Approach

Cast learning from language feedback as variational EM: the E-step actively trains the feedback-conditioned teacher with unpaired preference optimization (BCO) against a dynamic reference prior set to the current student (a sliding trust region); the M-step distills the refined teacher into the student on its own rollouts. Teacher and student share one network.

Key results

• On SciKnowEval with Qwen3-8B, VPD averages 77.15% vs. SDPO 74.44% and GRPO 73.11%.
• On LiveCodeBench v6 with Qwen3-8B (thinking off), VPD reaches 49.62% pass rate vs. SDPO 47.33% and GRPO 45.61%.
• On Math500 with Qwen3-8B trained on DAPO-Math, pure GRPO (83.8%) remains the dominant approach, outperforming VPD and SDPO.

Surprising findings

• Pure GRPO outperforms VPD and SDPO on mathematical reasoning and on base-model cold-start settings, so self-distillation from feedback does not dominate sparse RL there.
• Naively combining RL and self-distillation advantages (Advantage Reshaping/Reweighting) can degrade performance below pure SDPO.

Limitations

• VPD still lags pure GRPO on mathematical reasoning and base-model cold-start settings, which the authors attribute to noisy textual feedback.
• Shared-weight parameterization bounds teacher capacity to the student architecture.
• Self-critique results rely on the model as its own judge, which is a weaker signal than environment feedback.

Experimental setup

Datasets: SciKnowEval LiveCodeBench DAPO-Math

Benchmarks: SciKnowEval LiveCodeBench v6 Math500 AIME24 AIME25 AMC23

Metrics: accuracy pass rate

Baselines: GRPO SDPO SDPO+RL (Joint Loss) SDPO+RL (Adv Reshape) SDPO+RL (Adv Reweight)

Builds on: SDPO GRPO Binary Classifier Optimization (BCO) DPO RLSD

Closest papers by meaning

• KL for a KL: On-Policy Distillation with Control Variate Baseline (NeurIPS 2026 · similarity 0.90)
• The Many Faces of On-Policy Distillation: Pitfalls, Mechanisms, and Fixes (NeurIPS 2026 · similarity 0.90)
• Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation (NeurIPS 2026 · similarity 0.90)
• Entropy-Aware On-Policy Distillation of Language Models (ICML 2026 · similarity 0.90)
• Beyond Pass@ 1: Self-Play with Variational Problem Synthesis Sustains RLVR (ICLR 2026 · similarity 0.90)
• Your Teacher Can’t Help You Here: Combating Supervision Fidelity Decay in On-Policy Distillation (NeurIPS 2026 · similarity 0.89)