Research map / Large language models / Agents, code and math
Mixing Expert Knowledge: Bring Human Thoughts Back To the Game of Go
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mixed fine-tuning of Qwen2.5 base models on heuristic Go expert data plus long-CoT general reasoning data, followed by GRPO with a KataGo-annotated reward, yielding LoGos, a Go-playing LLM that keeps general reasoning ability.
Problem
General LLMs perform far below beginner level at Go, and domain expertise in scarce-corpus domains is hard to inject without large human reasoning annotation, while keeping general reasoning intact.
Approach
Inject expert Go knowledge via heuristic-template synthetic CoT data generated from KataGo annotations, mix it with general long-CoT data as a cold start, then apply GRPO with a segmented reward based on KataGo move rankings and win-rate accuracy so the model self-explores reasoning for next-move prediction.
Key results
- LoGos(32B) reaches 88.6% on KataGo-Bench-1K, versus 34.3% for Claude3.7-Sonnet and 87.8% for KataGo-HumanSL-9d.
- LoGos(7B) scores 88.1% on KataGo-Bench-1K and matches or beats DeepSeek-R1-Distill-Qwen-7B on most general benchmarks (e.g., KOR-Bench 65.7 vs 55.9, AIME 40.0 vs 33.3).
- Without cold start, RL from instruct and distilled models plateaus below beginner level (about 67%), and naive direct-prediction cold start caps performance below 50%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Mix structured Go expertise with general long-CoT data for cold-start fine-tuning, then apply GRPO | Qwen2.5-7B-Base and Qwen2.5-32B-Base · KataGo-Bench-1K | accuracy: 88.1; 88.6 | Helped |
| Apply GRPO directly without cold-start fine-tuning (ablation) | Qwen2.5 instruction models and DeepSeek-R1 distilled models · Go prediction | accuracy: below 67.36% | Hurt |
| Replace heuristic-rule dataset construction with direct next-move prediction for cold start (ablation) | 7B LoGos training · Go prediction | accuracy: below 50% compared with ... 88% | Hurt |
| Use only top-1 move rewards instead of the original reward design (ablation) | LoGos GRPO · Go prediction | performance | Hurt |
| Reward top-3 moves without distinguishing among them (ablation) | LoGos GRPO · Go prediction | performance | Hurt |
| Remove intra-level rewards by setting α1 and α2 to zero (ablation) | LoGos GRPO · Go prediction | performance | No clear effect |
| Add intra-level reward shaping to the tiered reward function (ablation) | LoGos GRPO · Go prediction | performance | Helped |
| Render the current board as a 2-D board state in the input (ablation) | LoGos · Go prediction on long game sequences | accuracy | Helped |
| Train cold start with 2 epochs rather than 1 epoch of Go data (ablation) | LoGos RL initialization · KataGo-Bench-1K | performance | Mixed |
| Increase Go data from none to 500K, 2M, 4M, or 10M examples (ablation) | LoGos general-reasoning model · MATH | score: 97.0; 97.1; 96.2; 97.4; 97.4 | No clear effect |
| Increase Go data from none to 500K, 2M, 4M, or 10M examples (ablation) | LoGos general-reasoning model · AIME | score: 73.2; 72.3; 72.5; 67.9; 72.9 | Mixed |
| Increase Go data from none to 500K, 2M, 4M, or 10M examples (ablation) | LoGos general-reasoning model · GPQA-Diamond | score: 61.9; 61.2; 60.1; 59.6; 60.6 | Hurt |
Surprising findings
- Reinforcement learning without the expert-data cold start plateaus lower for DeepSeek's R1-distilled models than for Qwen instruct models, so long-CoT data alone does not transfer to Go.
- The model achieves 96.5% accurate move predictions in human review but only 55.6% correct explanations, with frequent Go terminology misuse despite no terminology reward.
Limitations
- Requires a scalable external expert source (KataGo) and hand-designed heuristic templates, limiting transfer to other domains.
- RL training is slow and the autoregressive LLM is trained on far fewer game states than traditional Go AI.
- Context curse: accuracy declines as move-sequence length grows; partially mitigated with 2-D board rendering.
Experimental setup
Datasets: OpenThoughts-114K NuminaMath-QwQ-CoT-5M OpenCodeReasoning Bespoke-Stratos-17k AM-DeepSeek-R1-Distilled-1.4M KataGo-annotated Go game records (5M games, 10M states) Go commentary dataset (100K)
Benchmarks: KataGo-Bench-1K GPQA-Diamond BBEH KOR-Bench AIME MATH LiveCodeBench
Metrics: accuracy win rate Elo rating
Baselines: DeepSeek-R1 OpenAI-o1-mini Claude3.7-Sonnet Qwen2.5-7B-Instruct Qwen2.5-32B-Instruct DeepSeek-R1-Distill-Qwen-7B DeepSeek-R1-Distill-Qwen-32B KataGo-HumanSL Golaxy-GiantElephant
Builds on: GRPO DeepSeekMath KataGo AlphaZero VeRL
Closest papers by meaning
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.90)
- Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective (NeurIPS 2025 · similarity 0.89)
- Advancing LLM Reasoning with Natural Language and Numerical Feedback (ICML 2026 · similarity 0.89)
- KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation (NeurIPS 2025 · similarity 0.89)
- Distilling LLM Agent into Small Models with Retrieval and Code Tools (NeurIPS 2025 · similarity 0.89)
- Position: General Intelligence Requires Reward-based Pretraining (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.