Research map / Large language models / Agents, code and math
Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Flow of Reasoning (FoR), a GFlowNet-based trajectory-balance finetuning of LLMs on DAG-structured reasoning graphs that samples diverse high-reward reasoning paths from minimal (about 15-20) training examples.
Problem
Existing LLM multi-step reasoning methods optimize top-1 accuracy and yield few diverse valid solutions; SFT needs lots of labeled data and reward-maximizing RL collapses diversity.
Approach
Formulates multi-step LLM reasoning as a Markovian flow on a DAG and finetunes the LLM forward policy with a trajectory-balance GFlowNet objective (log-variance form), so trajectories are sampled proportional to reward, combined with off-policy replay, epsilon-sampling and a local search that destroys and reconstructs high-reward trajectories.
Key results
- On BlocksWorld 6-step, FoR reaches 78.44% accuracy and diversity 1.33 versus 34.68% for SFT and 69.70% for RAP.
- On Rubik's Cube FoR reaches 10.87% accuracy versus 4.92% for XoT, with creativity 82.61%.
- On PrOntoQA out-of-distribution, FoR achieves 63.07% prediction accuracy and 28.88% proof accuracy versus 50.10% and 24.60% for STaR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with FoR rather than RAP | Llama-3-8B · BlocksWorld 4-step | accuracy (%): 92.86 -> 98.41 (1.12) | Helped |
| Train with FoR rather than SFT | Llama-3-8B · Game24 | diversity: 1.18 -> 1.30 (0.09) | Helped |
| Train with FoR rather than XoT | Llama-3-8B · Rubik's Cube | accuracy (%): 4.92 -> 10.87 (1.18) | Helped |
| Train with FoR rather than CoT | Llama-3-8B · 1D-ARC | accuracy (%): 39.51 (1.94) -> 50.37 (1.60) | Helped |
| Add PRM to FoR (ablation) | FoR on Llama-3-8B · GSM8K | accuracy (%): 57.39 -> 62.62 | Helped |
| Add STaR to FoR (ablation) | FoR on Llama-3-8B · PrOntoQA in-distribution | proof accuracy (%): 54.60 (1.50) -> 54.70 (1.41) | No clear effect |
| Remove local search (ablation) | FoR on Llama-3-8B · BlocksWorld 6-step | accuracy (%): 78.44 (4.54) -> 53.90 (2.10) | Hurt |
| Add local search (ablation) | FoR without local search on Llama-3-8B · BlocksWorld 6-step | accuracy (%): 53.90 (2.10) -> 78.44 (4.54) | Helped |
| Remove augmented rewards (ablation) | FoR on Llama-3-8B · BlocksWorld 6-step | accuracy (%): 78.44 (4.54) -> 47.10 (1.30) | Hurt |
| Add augmented rewards (ablation) | FoR without augmented rewards on Llama-3-8B · BlocksWorld 6-step | accuracy (%): 47.10 (1.30) -> 78.44 (4.54) | Helped |
| Remove replay buffer (ablation) | FoR on Llama-3-8B · BlocksWorld 6-step | accuracy (%): 78.44 (4.54) -> 72.38 (1.71) | Hurt |
| Add replay buffer (ablation) | FoR without replay buffer on Llama-3-8B · BlocksWorld 6-step | accuracy (%): 72.38 (1.71) -> 78.44 (4.54) | Helped |
Surprising findings
- Diversity-encouraging prompts gave CoT and SFT baselines only modest gains, and FoR still outperformed them.
Limitations
- Training is considerably more expensive than SFT and PPO due to on-environment trajectory collection.
- Subtrajectory balance and other GFlowNet extensions are left for future work.
Experimental setup
Datasets: BlocksWorld LLM-reasoner Game24 dataset Rubik's Cube dataset 1D-ARC GSM8K PrOntoQA
Benchmarks: BlocksWorld Game of 24 Rubik's Cube 1D-ARC GSM8K PrOntoQA
Metrics: accuracy diversity creativity runtime
Baselines: CoT ToT RAP GoT XoT SFT PPO GFN-CoT STaR
Builds on: GFlowNet trajectory balance STaR RAP
Closest papers by meaning
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.90)
- SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.90)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.90)
- Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning (NeurIPS 2025 · similarity 0.90)
- Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.89)
- R-Diverse: Mitigating Diversity Illusion in Self-Play LLM Training (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.