Research map / Large language models / Instruction and fine-tuning

UFT: Unifying Supervised and Reinforcement Fine-Tuning

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Instruction data selection · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Unified Fine-Tuning (UFT), which combines GRPO-style RFT on hint-prefixed prompts with a log-likelihood term on the hint, and proves an exponential sample-complexity improvement over RFT in a tree model.

Problem

SFT overfits and limits larger models, while RFT depends heavily on base model strength and suffers sparse rewards for small models; RFT's sample complexity is exponential in reasoning length.

Approach

Unify SFT and RFT by starting rollouts from a prefix (hint) of the reference solution, annealing hint length to zero via a cosine-scheduled binomial sampler, and adding a log-likelihood term on the hint to the RL objective so the model memorizes solution prefixes while exploring the remainder.

Key results

• On Qwen2.5-0.5B, UFT averages 9.45% across nine benchmarks versus 3.25% for RFT and 7.28% for SFT-RFT.
• On Qwen2.5-3B, RFT averages 32.15% and UFT 30.93%, with SFT-RFT at 17.34%, so UFT roughly matches RFT on larger models.
• On Llama-3.2-1B, UFT averages 3.52% versus 0.80% for RFT and 2.49% for SFT.

Measured effects reported in the paper

Surprising findings

• On Qwen2.5-3B, the R^3 baseline (28.02%) and RFT (32.15%) outperform UFT on average, so UFT does not win at every scale.
• Llama-3.2-3B RFT averages 0.01% and SFT-RFT 4.79%, showing RFT alone can fail completely for some model-task pairs.

Limitations

• Uses only human-annotated solutions as supervision and GRPO as the RL algorithm.
• Experiments limited to models up to 3B due to computational constraints; larger-model claims are extrapolated.
• Theory is for a softmax tree model with deterministic transitions, not full LLM training.

Experimental setup

Datasets: Countdown MATH(3,4,5) Knights and Knaves logic puzzle (Logic)

Benchmarks: Countdown MATH(3,4,5) Logic AIME24 AMC MATH500 Minerva Olympiad GSM8k

Metrics: accuracy (pass@1) average accuracy

Baselines: SFT RFT SFT-RFT R^3

Builds on: GRPO R^3 (reverse curriculum reinforcement learning) TinyZero VERL

Closest papers by meaning

• The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models (NeurIPS 2025 · similarity 0.91)
• First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training (NeurIPS 2025 · similarity 0.90)
• Native Reasoning Models: Training Language Models to Reason on Unverifiable Data (ICLR 2026 · similarity 0.89)
• Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs (ICLR 2026 · similarity 0.89)
• AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy (ICLR 2026 · similarity 0.89)
• An Emulator for Fine-tuning Large Language Models using Small Language Models (ICLR 2024 · similarity 0.89)