Research map / Large language models / Reasoning and chain of thought

Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Webscale-RL, a pipeline that converts web-scale pretraining documents into 1.2M verifiable QA pairs for RL, and shows GRPO on Qwen2.5-3B beats continual pretraining and data refinement baselines with better token efficiency.

Problem

RL training is bottlenecked by small, low-diversity verifiable RL datasets (orders of magnitude smaller than pretraining corpora), limiting RL's scaling and general reasoning gains.

Approach

Convert pretraining documents into persona-driven, domain-conditioned, short-answer QA pairs grounded in the source text (rather than distilled from a strong teacher), with filtering and a verifier for correctness and leakage, producing a large diverse binary-reward RL dataset.

Key results

• Webscale-RL on Qwen2.5-3B averages 52.1 across seven benchmarks vs 48.7 for the strongest baseline (GDR) and 48.3 for continual pretraining.
• MATH500 rises from 47.6 (base) to 58.0 with Webscale-RL, close to Qwen2.5-7B's 60.8.
• RL with ~10M tokens matches continual pretraining with ~1B tokens on MMLU-Pro, claimed as over 100x data efficiency.

Surprising findings

• Continual-pretrained models often fail to answer instructions in zero-shot evaluation, so an SFT step was needed for fair comparison.
• Coding gains are small, attributed to low coding share in the pretraining source.

Limitations

• Coding domain underrepresented, yielding smaller gains on coding benchmarks.
• Binary reward relies on a generative LLM judge that adds substantial inference cost.
• Token-efficiency comparison depends on a custom token-accounting convention based on source documents.

Experimental setup

Datasets: DCLM Wikipedia MegaMath Stack-v2 OpenMathReasoning OpenCodeReasoning RedPajama-v2 FineWeb-Edu DCLM-baseline NaturalReasoning Nemotron DeepScaler

Benchmarks: MMLU-Pro BigBench GPQA-Diamond MATH500 GSM8K MBPP EvalPlus

Metrics: accuracy average score token efficiency

Baselines: Continual pretraining QuRating ProX Generative Data Refinement (GDR) Qwen2.5-7B

Builds on: GRPO DeepScaleR OpenR1-Math VeRL NaturalReasoning

Closest papers by meaning

• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.89)
• Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text (ICML 2026 · similarity 0.89)
• MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions (ICLR 2025 · similarity 0.89)
• d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay (NeurIPS 2025 · similarity 0.88)
• RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning (NeurIPS 2025 · similarity 0.88)