Research map / Large language models / Instruction and fine-tuning
The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UPFT, an unsupervised fine-tuning method that trains only on the first few tokens (as few as 8) of single sampled responses, plus a small fraction of full-trace unsupervised SFT, matching supervised RFT at a fraction of the cost.
Problem
Improving LLM reasoning typically requires labeled data or costly sampling-and-filtering (rejection sampling fine-tuning), which is expensive and needs ground-truth verification.
Approach
Observes "prefix self-consistency": solution trajectories for the same question share initial reasoning tokens, and errors concentrate in later steps. Fine-tunes only on short prefixes of one unfiltered sample per question (plus a small share of full unsupervised traces for structure), removing the need for labels or rejection sampling.
Key results
- Qwen2.5-Math-7B-Instruct with U-Hard: UPFT reaches 54.5% average vs 51.3% for unsupervised SFT; with DeepSeek-R1-Distill-Qwen-7B, UPFT reaches 61.6% vs 56.4% for SFT.
- Supervised setting: UPFT matches RFT on Qwen2.5-Math-7B-Instruct (52.6% avg) using 0.6M vs 51.7M sampling tokens; on DeepSeek-R1-Distill-Qwen-7B UPFT reaches 58.7% vs RFT 57.2%.
- UPFT with label filtering reaches 58.8% average on DeepSeek-R1-Distill-Qwen-7B, the best among compared methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on prefixes instead of full generated solutions | Qwen2.5-Math-7B-Instruct with U-Hard, unsupervised sampling · Average across benchmarks | accuracy: 51.3 -> 54.5 | Helped |
| Train on prefixes instead of full generated solutions | Qwen2.5-Math-7B-Instruct with U-Hard, unsupervised sampling · AIME2024 | accuracy: 16.7 -> 26.6 | Helped |
| Train on prefixes instead of full generated solutions | DeepSeek-R1-Distill-Qwen-7B with U-Hard, unsupervised sampling · Average across benchmarks | accuracy: 56.4 -> 61.6 | Helped |
| Train on prefixes instead of full generated solutions | DeepSeek-R1-Distill-Qwen-7B with U-Hard, unsupervised sampling · AIME2024 | accuracy: 36.7 -> 50.0 | Helped |
| Train on prefixes rather than RFT-selected full solutions | Qwen2.5-Math-7B-Instruct, supervised sampling · Average across benchmarks | accuracy: 52.6 vs. 52.6 | No clear effect |
| Train on prefixes rather than RFT-selected full solutions | Llama-3.1-8B-Instruct, supervised sampling · Average across benchmarks | accuracy: 38.3 vs. 38.5 | No clear effect |
| Train on prefixes rather than RFT-selected full solutions | DeepSeek-R1-Distill-Qwen-7B, supervised sampling · Average across benchmarks | accuracy: 57.2 -> 58.7 | Helped |
| Add label filtering to UPFT | Llama-3.1-8B-Instruct · Average across benchmarks | accuracy: 38.3 -> 38.8 | Helped |
| Add label filtering to UPFT | DeepSeek-R1-Distill-Qwen-7B · Average across benchmarks | accuracy: 58.7 -> 58.8 | No clear effect |
| Use an 8-token prefix (ablation) | UPFT with Llama-3.1-8B-Instruct · MATH500 test set | accuracy: 52.0% | Helped |
| Vary prefix length from 8 to 32 tokens (ablation) | UPFT with Qwen2.5-Math-7B-Instruct · MATH500 test set | accuracy: 8 to 32 tokens | No clear effect |
| Increase the structure tuning ratio up to 10% (ablation) | UPFT with Llama-3.1-8B-Instruct and Qwen2.5-Math-7B-Instruct · MATH500 test set | accuracy: p=10% | Helped |
Surprising findings
- Training on only the first 8 tokens of Llama-3.1-8B-Instruct responses gives the best MATH500 accuracy, with prefix length less sensitive for the math-specialized model.
- Unsupervised UPFT on DeepSeek-R1-Distill-Qwen-7B with 561-token average samples outperforms SFT trained on 3,440-token samples.
Limitations
- Gains over RFT are small or absent on some models (e.g., Llama average 38.3 vs RFT 38.5), and differences are often within a few points on small test sets like AIME2024 (30 problems).
- Optimal prefix length and structure-tuning ratio are model-specific and tuned on the MATH500 test set via ablations.
- U-Hard dataset is not released in the text, limiting reproducibility.
Experimental setup
Datasets: PRM800K-derived MATH problems (PRM-12K) OpenMathInstruct2 (OMI2-600K) LIMO U-Hard
Benchmarks: GSM8K MATH500 AIME2024 GPQA Diamond
Metrics: accuracy average accuracy sampling tokens tuning tokens
Baselines: SFT RFT V-STaR Label Filter
Builds on: rejection sampling fine-tuning STaR V-STaR LIMO Omni-MATH
Closest papers by meaning
- UFT: Unifying Supervised and Reinforcement Fine-Tuning (NeurIPS 2025 · similarity 0.91)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.90)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.90)
- LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning (ICML 2025 · similarity 0.90)
- THE PATH OF LEAST RESISTANCE: GUIDING LLM REASONING TRAJECTORIES WITH PREFIX CONSENSUS (ICLR 2026 · similarity 0.89)
- A State-Transition Framework for Efficient LLM Reasoning (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.