Research map / Reinforcement learning / RL from feedback
SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Compares SFT and RL post-training of Llama-3.2-Vision-11B on rule-based and visual generalization using the GeneralPoints card game and V-IRL navigation, finding RL generalizes while SFT memorizes.
Problem
Unclear whether SFT or RL primarily memorizes training data versus learning generalizable rules, across textual rule variants and visual variants in foundation model post-training.
Approach
Use controlled rule and visual variants of GeneralPoints (arithmetic card game) and V-IRL (navigation) to separate memorization from generalization, training SFT and multi-turn PPO-based RL with outcome rewards and sequential revision, then compare OOD performance under equal compute.
Key results
- RL improves OOD performance on GP-L from 11.5% to 15.0% while SFT drops it to 3.4%.
- On V-IRL-L, RL raises OOD from 80.8% to 91.8% while SFT falls to 1.3%.
- Multi-turn RL reaches 77.8% on the V-IRL VLN mini benchmark versus previous SOTA 44.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Post-train with RL instead of the shared initial checkpoint | Llama-3.2-Vision-11B · GeneralPoints-L rule OOD | success rate: 11.5% -> 15.0% | Helped |
| Post-train with SFT instead of the shared initial checkpoint | Llama-3.2-Vision-11B · GeneralPoints-L rule OOD | success rate: 11.5% -> 3.4% | Hurt |
| Post-train with RL instead of the shared initial checkpoint | Llama-3.2-Vision-11B · V-IRL-L rule OOD | per-step accuracy: 80.8% -> 91.8% | Helped |
| Post-train with SFT instead of the shared initial checkpoint | Llama-3.2-Vision-11B · V-IRL-L rule OOD | per-step accuracy: 80.8% -> 1.3% | Hurt |
| Post-train with RL instead of the shared initial checkpoint | Llama-3.2-Vision-11B · GeneralPoints-VL rule OOD | success rate: 11.2% -> 14.2% | Helped |
| Post-train with SFT instead of the shared initial checkpoint | Llama-3.2-Vision-11B · GeneralPoints-VL rule OOD | success rate: 11.2% -> 5.6% | Hurt |
| Post-train with RL instead of the shared initial checkpoint | Llama-3.2-Vision-11B · V-IRL-VL rule OOD | per-step accuracy: 35.7% -> 45.0% | Helped |
| Post-train with SFT instead of the shared initial checkpoint | Llama-3.2-Vision-11B · V-IRL-VL rule OOD | per-step accuracy: 35.7% -> 2.5% | Hurt |
| Post-train with RL instead of the shared initial checkpoint | Llama-3.2-Vision-11B · GeneralPoints-VL visual OOD | success rate: 23.6% -> 41.2% | Helped |
| Post-train with SFT instead of the shared initial checkpoint | Llama-3.2-Vision-11B · GeneralPoints-VL visual OOD | success rate: 23.6% -> 13.7% | Hurt |
| Post-train with RL instead of the shared initial checkpoint | Llama-3.2-Vision-11B · V-IRL-VL visual OOD (VLN mini benchmark) | success rate: 16.7% -> 77.8% | Helped |
| Post-train with SFT instead of the shared initial checkpoint | Llama-3.2-Vision-11B · V-IRL-VL visual OOD (VLN mini benchmark) | success rate: 16.7% -> 11.1% | Hurt |
Surprising findings
- Without SFT initialization, end-to-end RL fails to improve because the base Llama-3.2-Vision-11B does not follow instructions.
- RL cannot recover OOD performance from an overly overfitted SFT checkpoint.
Limitations
- SFT fails to reach comparable in-distribution performance with RL on GP-VL, left unresolved.
- RL has limited effectiveness from extremely underfit or overfit SFT checkpoints.
- Experiments limited to one backbone model (Llama-3.2-Vision-11B) and two synthetic/simulated tasks.
Experimental setup
Datasets: GeneralPoints card quadruples sampled from 52-card deck V-IRL routes (1000 NYC routes) V-IRL VLN mini benchmark
Benchmarks: GeneralPoints (GP-L, GP-VL) V-IRL (V-IRL-L, V-IRL-VL) V-IRL VLN mini benchmark
Metrics: success rate per-step accuracy
Baselines: SFT Init checkpoint RL4VLM
Builds on: RL4VLM PPO V-IRL Points24
Closest papers by meaning
- Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead (ICLR 2026 · similarity 0.91)
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.90)
- AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy (ICLR 2026 · similarity 0.90)
- What Can RL Bring to VLA Generalization? An Empirical Study (NeurIPS 2025 · similarity 0.90)
- Generalization of RLVR Using Causal Reasoning as a Testbed (ICLR 2026 · similarity 0.90)
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.