Research map / Reinforcement learning / RL from feedback
UFO-RL: Uncertainty-Focused Optimization for Efficient Reinforcement Learning Data Selection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UFO-RL, which selects the 10% of RL training prompts with intermediate model confidence (average log-probability from a single forward pass, mapped to a fuzziness score), matching or beating full-data GRPO training at up to 16x lower training time.
Problem
RL fine-tuning of LLMs is costly because multi-sampling is needed to estimate per-instance difficulty for data selection, and existing filters (e.g., DAPO's accuracy filter) still require that multi-sampling while removing only a small share of data.
Approach
Use a single-pass confidence score (exponentiated mean token log-probability of a generated answer) as a cheap difficulty proxy, and select the training prompts whose confidence is closest to the dataset mean (a Zone-of-Proximal-Development-inspired "fuzziness" score) rather than the easiest or hardest ones.
Key results
- Training on a 10% UFO-selected subset matches or exceeds full-data RL on most settings, e.g., Qwen2.5-3B GSM8K 84.37 vs 83.33 full data when trained on GSM8K, and Math500 67.40 vs 66.20.
- Confidence scoring is reported up to 185x faster than 16-sample accuracy estimation (Qwen2.5-0.5B: 18s vs 3337s on one A100).
- RL training on the 10% UFO subset takes 11x-16x less wall-clock time than full-data training (e.g., Mistral-7B 1454s vs 22955s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select the 10% mid-confidence subset instead of using all GSM8K training data | Qwen2.5-3B · GSM8K test (trained on GSM8K) | accuracy (%): 83.33 -> 84.37 | Helped |
| Select the 10% mid-confidence subset instead of using all GSM8K training data | Qwen2.5-7B · Math500 (trained on GSM8K) | accuracy (%): 75.00 -> 76.40 | Helped |
| Select the 10% mid-confidence subset instead of using all GSM8K training data | LLaMA3.1-8B · GSM8K test (trained on GSM8K) | accuracy (%): 88.62 -> 88.30 | No clear effect |
| Select the 10% mid-confidence subset instead of using all GSM8K training data | Mistral-7B · GSM8K test (trained on GSM8K) | accuracy (%): 53.87 -> 56.67 | Helped |
| Use single-pass confidence instead of multi-sample accuracy for data evaluation | Qwen2.5-0.5B · GSM8K training set | evaluation time: 3337s -> 18s; × 185 | Helped |
| Use single-pass confidence instead of multi-sample accuracy for data evaluation | Qwen2.5-1.5B · GSM8K training set | evaluation time: 3712s -> 45s; × 82 | Helped |
| Use single-pass confidence instead of multi-sample accuracy for data evaluation | Qwen2.5-3B · GSM8K training set | evaluation time: 4186s -> 89s; × 47 | Helped |
| Use single-pass confidence instead of multi-sample accuracy for data evaluation | Qwen2.5-7B · GSM8K training set | evaluation time: 6827s -> 175s; × 39 | Helped |
| Train on the highest-confidence 10% rather than the random 10% subset | Qwen2.5-7B · GSM8K test (trained on GSM8K) | accuracy (%): 90.93 -> 71.09 | Hurt |
| Train on the lowest-confidence 10% rather than the random 10% subset | Qwen2.5-0.5B · GSM8K test (trained on GSM8K) | accuracy (%): 41.96 -> 41.57 | No clear effect |
| Remove examples with 0% or 100% multi-sample accuracy | Qwen2.5-3B · GSM8K test (trained on GSM8K) | accuracy (%): 83.33 -> 83.61 | Helped |
Surprising findings
- Training only on the highest-confidence 10% (High Conf) causes large drops for some models, e.g., Qwen2.5-3B GSM8K falls to 62.67 versus 83.33 for full data.
- On DAPO-MATH-17K, Qwen2.5-0.5B full-data training collapses to 12.44 GSM8K accuracy while the UFO subset reaches 41.43.
Limitations
- Experiments limited to models up to 8B parameters and training sets up to 17K examples.
- Confidence is computed from the model's own generated answer, so the score is model-specific and may not transfer across models.
- MMLU results show marginal differences across selection strategies, so the gains are mainly on math benchmarks.
Experimental setup
Datasets: GSM8K DAPO-MATH-17K Math500 MMLU
Benchmarks: GSM8K Math500 MMLU
Metrics: accuracy training time (s) speedup
Baselines: Full Data High Conf Low Conf Random Acc_Filter (DAPO-style accuracy filter)
Builds on: GRPO DAPO Open-R1 LIMR
Closest papers by meaning
- Efficient Reinforcement Learning with Large Language Model Priors (ICLR 2025 · similarity 0.89)
- Accelerating RL for LLM Reasoning with Optimal Advantage Regression (NeurIPS 2025 · similarity 0.89)
- ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.89)
- On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.89)
- Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection (ICLR 2026 · similarity 0.88)
- Diversity-Aware Policy Optimization for Large Language Model Reasoning (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.