Research map / Large language models / Reasoning and chain of thought
B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasoners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes exploration and exploitation dynamics in iterative self-improvement (online RFT) for reasoning and proposes B-STaR, which adapts sampling temperature and reward threshold per iteration to maximize a balance score.
Problem
Iterative self-improvement for reasoning saturates after a few iterations, and the factors behind this (declining exploration and weakening reward-based exploitation) are poorly understood and not monitored.
Approach
Define exploration (Pass@K-S, diversity) and exploitation (Reward@K-S) metrics and a balance score that rewards many unique correct samples with high correct ratio; then automatically choose sampling temperature and reward threshold each iteration to maximize the average balance score on a small query subset.
Key results
- On Mistral-7B, B-STaR reaches 53.8 Pass@1 on GSM8K and 27.8 on MATH, versus 46.8 and 23.2 for Online RFT (w/ RM).
- On Llama-3.1-8B, B-STaR reaches 61.6 GSM8K, 29.2 MATH, 18.1 APPS, and 86.3 ARC-C Pass@1, the best among compared methods.
- Ablation: temperature-only adjustment gives 53.1 GSM8K / 25.0 MATH and threshold-only gives 49.1 / 24.6, both below the combined 53.8 / 27.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| dynamically balance sampling temperature and reward threshold | Online RFT with Mistral-7B · GSM8K | Pass@1: 46.8 -> 53.8 | Helped |
| dynamically balance sampling temperature and reward threshold | Online RFT with Mistral-7B · MATH | Pass@1: 24.4 -> 27.8 | Helped |
| dynamically balance sampling temperature and reward threshold | Online RFT with Llama-3-8B · APPS | Pass@1: 17.3 -> 19.6 | Helped |
| dynamically balance sampling temperature and reward threshold | Online RFT with Mistral-7B · ARC-Challenge | Pass@1: 71.2 -> 73.0 | Helped |
| add dynamic temperature adjustment while leaving threshold at its default (ablation) | Online RFT with Mistral-7B · GSM8K | Pass@1: 46.8 -> 53.1 | Helped |
| add dynamic temperature adjustment while leaving threshold at its default (ablation) | Online RFT with Mistral-7B · MATH | Pass@1: 23.2 -> 25.0 | Helped |
| add dynamic reward-threshold adjustment while leaving temperature at its default (ablation) | Online RFT with Mistral-7B · GSM8K | Pass@1: 46.8 -> 49.1 | Helped |
| add dynamic reward-threshold adjustment while leaving temperature at its default (ablation) | Online RFT with Mistral-7B · MATH | Pass@1: 23.2 -> 24.6 | Helped |
| add both dynamic temperature and reward-threshold adjustment instead of temperature adjustment alone (ablation) | B-STaR with Mistral-7B · GSM8K | Pass@1: 53.1 -> 53.8 | Helped |
| add both dynamic temperature and reward-threshold adjustment instead of temperature adjustment alone (ablation) | B-STaR with Mistral-7B · MATH | Pass@1: 25.0 -> 27.8 | Helped |
| add process reward model to answer matching | Online RFT with Mistral-7B · GSM8K | Pass@1: 44.0 -> 46.8 | Helped |
| add process reward model to answer matching | Online RFT with Mistral-7B · MATH | Pass@1: 23.0 -> 23.2 | No clear effect |
Surprising findings
- Diversity and Pass@K-S decline over iterations under answer-only filtering, while adding a PRM reward helps retain exploration.
- Fixing the Online RFT configuration at the B-STaR final values (temp 1.1, threshold -0.1) lowers performance (40.4 GSM8K, 18.2 MATH), indicating static configurations do not transfer across iterations.
Limitations
- Only temperature and reward threshold are adjusted; sample size is fixed to the maximum budget.
- The PRM is a 7B model and is less discriminative on harder MATH problems, limiting exploitation gains.
- Exploration and exploitation are studied mainly on math tasks; coding and commonsense experiments use binary rewards only.
Experimental setup
Datasets: MATH GSM8K APPS ARC-Challenge
Benchmarks: MATH500 GSM8K APPS ARC-Challenge
Metrics: Pass@1 Pass@32 Pass@32-4 Pass@K-S Reward@K-S Distinct Equations diversity balance score
Baselines: SFT Rest-EM Iterative RFT Online RFT
Builds on: STaR ReST-EM Rejection sampling fine-tuning Online RFT MATH-Shepherd
Closest papers by meaning
- A Task-centric Theory for Iterative Self-Improvement with Easy-to-Hard Curricula (ICML 2026 · similarity 0.90)
- Better, Faster: Harnessing Self-Improvement in Large Reasoning Models (ICML 2026 · similarity 0.88)
- Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability (ICML 2026 · similarity 0.88)
- Progress or Regress? Self-Improvement Reversal in Post-training (ICLR 2025 · similarity 0.88)
- Rethinking Thinking Tokens: LLMs as Improvement Operators (ICML 2026 · similarity 0.87)
- ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.