Research map / Large language models / Retrieval and knowledge
Position: General Intelligence Requires Reward-based Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing general intelligence needs reward-based (RL) pretraining, a synthetic-task reasoning prior, and decoupled memory/reasoning architectures, supported by esoteric-language evaluations, a Go 9x9 AlphaZero-vs-AlphaGo-style comparison, and a small Qwen2.5-1.5B SFT-vs-RFT experiment.
Problem
LLMs pretrained with next-token prediction on passive data reason in ways that overfit to training data and transfer poorly to novel contexts, such as unfamiliar programming syntaxes, so they fall short of AGI.
Approach
Replace supervised next-token pretraining with reward-based pretraining from scratch, bootstrap exploration with synthetic tasks that yield a transferable reasoning prior, and decouple knowledge (external memory) from reasoning (small context window with learned read/write via RL).
Key results
- o1 reached 0.71 accuracy on Brainf**k print (1 ICL example) and 0.95 on copy, while other models scored near 0 on sorting; o1 scored 0.01 on sorting.
- On the orthogonality task, pure RFT on Qwen2.5-1.5B-Instruct reached 93% test accuracy versus 80% for SFT-then-RFT, which hit 100% train accuracy.
- In 9x9 Go, the RPT agent won 100% against the SPT agent and 66% (KL 0.1) and 92% (KL 0.5) against SPT+RFT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RFT without first applying SFT on passive demonstrations | Qwen2.5-1.5B-Instruct · Orthogonality task test | accuracy: 80 to 93 | Helped |
| Apply SFT before RFT instead of using RFT alone | Qwen2.5-1.5B-Instruct · Orthogonality task train | accuracy: 94 to 100 | Helped |
| Increase SFT training from 20 to 100 epochs (ablation) | Qwen2.5-1.5B-Instruct · Orthogonality task test | accuracy: 62.0 to 82.0; Train Acc CoT: 74.0 to 100.0 | Mixed |
| Increase in-context examples from 1 to 10 | Llama 3.1 8B · Brainf**k Print | accuracy: 0.01 -> 0.07 | Helped |
| Increase in-context examples from 1 to 10 | Llama 3.1 8B · Befunge Print | accuracy: 0.19 -> 0.72 | Helped |
| Use reward-based pretraining rather than SPT on expert demonstrations | Go 9×9 · Go 9×9 vs SPT | win rate: 100% | Helped |
| Use reward-based pretraining rather than SPT-then-RFT with KL coefficient 0.1 | Go 9×9 · Go 9×9 vs SPT-then-RFT (KL coefficient 0.1) | win rate: 66% | Helped |
| Use reward-based pretraining rather than SPT-then-RFT with KL coefficient 0.5 | Go 9×9 · Go 9×9 vs SPT-then-RFT (KL coefficient 0.5) | win rate: 92% | Helped |
| Increase the KL constraint from 0.1 to 0.5 (ablation) | SPT-then-RFT Go agent · Go 9×9 against RPT | RPT win rate: 66% to 92% | Helped |
Surprising findings
- Brainf**k performance barely improved (about 4% on average) when in-context examples went from 1 to 10.
- SFT-then-RFT models overfit to the training distribution, scoring lower on test than pure RFT despite higher training accuracy.
Limitations
- Core proposals (RL pretraining from scratch, synthetic reasoning prior, memory architecture) are untested at scale and remain hypotheses.
- RL-driven training is computationally expensive and exploration in natural language is very large.
- Experiments are small: 100 training examples for the LLM SFT/RFT test and a simplified 9x9 Go setting.
Experimental setup
Datasets: Brainf**k and Befunge algorithmic tasks synthetic vector orthogonality task professional 9x9 Go game records (GoQuest and other archives)
Benchmarks: Brainf**k print/sort/copy tasks Befunge print/Fibonacci/factorial tasks synthetic vector orthogonality task Go 9x9 win rate
Metrics: accuracy win rate
Baselines: Llama 3.1 8B Llama 3.1 70B Qwen2.5 Coder 7B Qwen2.5 Coder 32B GPT-4o o1 SFT-then-RFT
Builds on: AlphaZero AlphaGo LoRA RLOO Chain-of-Thought
Closest papers by meaning
- General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.91)
- Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models (ICLR 2025 · similarity 0.90)
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.89)
- Eliciting Reasoning in Language Models with Cognitive Tools (NeurIPS 2025 · similarity 0.89)
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.89)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.