Research map / Large language models / Reasoning and chain of thought

Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that high post-SFT scores often fail to predict RLVR outcomes for math reasoning and proposes validation generalization loss and large-k Pass@k as better predictors of post-RL performance.

Problem

Whether models with better SFT performance always end up better after RLVR, and how to predict which SFT checkpoint or data recipe will yield the best post-RL outcome without running expensive RL.

Approach

Identifies failure modes of SFT-metric-based selection (over-epoching, short/simple examples) and proposes two pre-RL predictors: SFT validation generalization loss (turning point of overfitting) and Pass@k at large k (k=64), which correlate better with post-RL Pass@1 than post-SFT Pass@1.

Key results

• Generalization loss plus Pass@64 averages Spearman 0.97 on Llama3-8B for dataset-level prediction vs 0.75 for post-SFT Pass@1 baseline.
• Pass@64 raises instance-level Spearman from 0.69 to 0.94 (Llama3-8B) and from 0.70 to 0.98 (Mistral-NeMo-12B).
• Over-epoched SFT gains +8.75% on MATH-500 yet yields -1.43% smaller RL improvement than non-repeated data on Llama3-8B.

Surprising findings

• Models with the best post-SFT scores (e.g., more epochs, shortest examples) often end up with lower final RL performance.
• On Qwen3-4B-base, post-SFT performance appears uncorrelated with post-RL performance, and state-of-the-art SFT data give little benefit after RL.

Limitations

• Study limited to math reasoning and online GRPO; other RL methods, DPO, and other domains not tested.
• Predicting Pass@large k requires many samples per problem, which is expensive with long sequences.
• Qwen3 results show little difference across SFT recipes, so they are only qualitative.

Experimental setup

Datasets: Llama-Nemotron-SFT AceReasoner1.1-SFT OpenR1-Math MATH train split DeepScaleR

Benchmarks: MATH-500 AIME 1983-2024 GSM8k AIME 2025 AMC Olympiad Minerva

Metrics: Pass@1 averaged over 64 repetitions Pass@64 coefficient of determination (R^2) Spearman rank correlation validation generalization loss

Baselines: post-SFT Pass@1 selection

Builds on: GRPO DeepSeek-R1 DeepScaleR LLaMA-Factory verl

Closest papers by meaning

• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.93)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.91)
• AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy (ICLR 2026 · similarity 0.91)
• First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training (NeurIPS 2025 · similarity 0.91)
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training (ICML 2025 · similarity 0.91)
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.90)