Research map / Large language models / Reasoning and chain of thought
BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BRiTE, an EM-style probabilistic framework where RL (PPO/GRPO) with a log-probability reward-shaping scheme approximates the posterior over reasoning traces, followed by SFT or DPO on the bootstrapped rationales, improving math and code reasoning without human-annotated rationales.
Problem
LLM reasoning reliability depends on high-quality thinking processes (CoT rationales), which are costly to annotate and hard to generate automatically; existing rejection-sampling EM methods do not optimize the rationale generator itself.
Approach
Model the prompt, latent thinking process, response and evaluation signal as a probabilistic graphical model; cast posterior sampling of correct rationales as an entropy-regularized MDP solved by RL, with per-token reward given by log-probabilities (reward shaping), then fine-tune the LLM on the sampled rationales (EM-style M-step).
Key results
- On GSM8K with Gemma-1.1-7B-it, BRiTE reaches 59.2 vs 58.4 for RS and 57.5 for human-annotated SFT; on MATH with Gemma-1.1-7B-it it reaches 23.7 vs 18.7 for RS.
- With Qwen2.5-7B trained on a ~40K mixed math set, BRiTE (psi-update) reaches 79.1 on MATH500 vs 54.3 for RS and 44.1 for the base model.
- BRiTE-iter-2 (psi-update) reaches 80.6 on MATH500 and 41.3 on Minerva Math, the best reported numbers in that table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BRiTE instead of SFT | Mistral-7B-Instruct-v0.2 · GSM8K | accuracy: 52.8 -> 52.2 | Hurt |
| Use BRiTE instead of SFT | Mistral-7B-Instruct-v0.2 · MATH | accuracy: 13.6 -> 11.2 | Hurt |
| Use BRiTE instead of rejection sampling | Gemma-1.1-7B-it · GSM8K | accuracy: 58.4 -> 59.2 | Helped |
| Use BRiTE instead of SFT | Gemma-1.1-7B-it · MATH | accuracy: 19.6 -> 23.7 | Helped |
| Use BRiTE instead of rejection sampling | Gemma-2-9B-it · GSM8K | accuracy: 87.6 -> 89.7 | Helped |
| Use BRiTE instead of rejection sampling | Gemma-2-9B-it · MATH | accuracy: 47.5 -> 50.5 | Helped |
| Use BRiTE instead of rejection sampling | Llama-3-8B-Instruct · GSM8K | accuracy: 79.5 -> 81.0 | Helped |
| Use BRiTE instead of rejection sampling | Llama-3-8B-Instruct · MATH | accuracy: 28.9 -> 30.0 | Helped |
| Use BRiTE-iter-2 ($\psi$-update) instead of RS | Qwen2.5-7B · MATH500 | pass@1 accuracy: 54.3 -> 80.6 | Helped |
| Use BRiTE-iter-2 ($\psi$-update) instead of RS | Qwen2.5-7B · Minerva Math | pass@1 accuracy: 21.0 -> 41.3 | Helped |
| Use BRiTE-iter-2 ($\theta$-update) instead of RS | Qwen2.5-7B · OlympiadBench | pass@1 accuracy: 23.1 -> 37.9 | Helped |
| Use BRiTE-iter-2 ($\theta$-update) instead of RS | Qwen2.5-7B · AIME24 | pass@1 accuracy: 5.6 -> 15.3 | Helped |
Surprising findings
- BRiTE SFT on RL-generated rationales matches or exceeds SFT on human-annotated rationales on GSM8K and MATH without any human rationales.
- Training with BRiTE reportedly yields only small gains over rejection sampling on instruct models, since the base models are already post-trained and the dataset is small.
Limitations
- Theoretical convergence result relies on a concavity assumption and an RKHS parametrization of logits that only loosely matches transformer LLMs.
- Gains over RS are mixed in the instruct-model experiments (e.g., BRiTE slightly below SFT on Mistral GSM8K and on Gemma-2-9B-it GSM8K relative to RS table entries), and the paper notes limited advantage when base models are already post-trained.
- Iterative training plateaus after the first iteration in the Qwen experiments.
Experimental setup
Datasets: GSM8K MATH RUC-AIBOX/STILL-3-Preview-RL-Data OpenCoder-LLM/opc-sft-stage2 RLHF4MATH prompt_iter1,2,3
Benchmarks: GSM8K MATH MATH500 Minerva Math OlympiadBench AIME24 AMC23 GPQA Diamond
Metrics: accuracy pass@1
Baselines: rejection sampling (RS) SFT with human-annotated rationales iterative DPO
Builds on: expectation-maximization rejection sampling fine-tuning (ReST-EM) PPO iterative DPO STaR
Closest papers by meaning
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.89)
- T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.89)
- Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization (ICLR 2026 · similarity 0.89)
- Thinker: Learning to Think Fast and Slow (NeurIPS 2025 · similarity 0.88)
- h1: Bootstrapping LLMs to Reason over Longer Horizons via Reinforcement Learning (ICML 2026 · similarity 0.88)
- Learning to Think: Information-Theoretic Reinforcement Fine-Tuning for LLMs (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.