Research map / Large language models / Reasoning and chain of thought

BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BOTS, a Bayesian online task selection framework for LLM reinforcement finetuning that fuses explicit rollout evidence with interpolation-based implicit difficulty estimates via Thompson sampling.

Problem

Online task selection in LLM reinforcement finetuning is costly and suboptimal: uniform sampling wastes compute on trivial or unsolvable tasks, while existing methods either need oversampling rollouts or use only one evidence source.

Approach

Model task difficulty as Beta posteriors updated with both explicit success/failure counts from selected tasks and pseudo-counts for unselected tasks from a lightweight interpolation between weak and strong reference models; select tasks via Thompson sampling near a target success rate of 0.5.

Key results

• On Qwen2.5-1.5B math, BOTS reaches TTB(100%)=0.72 (about 28% fewer steps) versus random baseline.
• On Qwen2.5-7B logic, BOTS achieves TTB(100%)=0.50, a 50% acceleration over random.
• For Qwen2.5-1.5B across six metrics per domain, BOTS earns 10 first-place and 6 second-place finishes out of 18 metrics.

Surprising findings

• Relying solely on implicit evidence (rho=1) degrades long-term performance as selection drifts toward tasks with success probability 1.

Limitations

• Focused on binary-reward tasks; non-binary rewards not validated.
• Update hyperparameters (lambda, rho) are fixed rather than adaptive across training stages.
• Implicit evidence requires reference-model scores; one-time rollout cost otherwise, and extrapolation can reduce accuracy.

Experimental setup

Datasets: GURU Skywork OR1 DAPO DeepScaler LeetCode TACO-verified PrimeIntellect LiveCodeBench ARC-AGI ARC-AGI-2 BARC

Benchmarks: MATH500 AMC23 AIME24 HumanEval MBPP LiveCodeBench Ordering Puzzles ARC-AGI

Metrics: Effective Task Ratio Time-to-Baseline Best-so-far

Baselines: Random Offline curriculum MoPPS DOTS (proxy) Dynamic Sampling

Builds on: GRPO MoPPS DOTS Thompson sampling

Closest papers by meaning

• Vanishing Gradients in Reinforcement Finetuning of Language Models (ICLR 2024 · similarity 0.90)
• Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models (ICLR 2026 · similarity 0.89)
• TTRL: Test-Time Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs (NeurIPS 2025 · similarity 0.89)
• Efficient Reinforcement Learning with Large Language Model Priors (ICLR 2025 · similarity 0.89)
• Can large language models explore in-context? (NeurIPS 2024 · similarity 0.89)