Research map / Reinforcement learning / RL from feedback

Finetuning with Sampling: Make SFT Generalize, Not Forget

NeurIPS 2026 · Reinforcement learning › RL from feedback · approach: RL fine-tuning of generative models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

The paper examines why SFT is thought to generalize poorly and forget, and proposes changes to finetuning with sampling so SFT generalizes without forgetting.

Problem

Supervised finetuning of frontier models is believed to generalize weakly and cause forgetting, while on-policy RL is preferred for acquiring new capabilities without losing existing ones.

Approach

Reframes SFT with sampling so that finetuning on self-generated or sampled outputs can retain the benefits of SFT (learning from demonstrations) while mitigating its generalization and forgetting problems. Details are not available beyond the abstract.

Experimental setup

Baselines: supervised finetuning on-policy reinforcement learning

Closest papers by meaning

• Towards On-Policy SFT: Distribution Discriminant Theory and its Applications in LLM Training (ICML 2026 · similarity 0.91)
• Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning (NeurIPS 2021 · similarity 0.90)
• Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.88)
• Reasoning with Sampling: Your Base Model is Smarter Than You Think (ICLR 2026 · similarity 0.87)
• Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards (NeurIPS 2025 · similarity 0.87)
• Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.87)