Research map / Large language models / Reasoning and chain of thought

Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Synthesizes over 0.7M multiple-choice fill-in-the-middle RLVR tasks from unverifiable internet text (textbooks, AoPS, coding solutions, FineWeb cyber scrapes) with GPT-5, and uses them to revive RL training saturated on ProRL data.

Problem

Scaling RLVR is bottlenecked by limited verifiable data; existing RL gains saturate, and many reasoning-rich corpora (textbooks, proofs, coding problems without tests) are unusable under standard verifiers.

Approach

Convert unverifiable reasoning text into MCQ fill-in-the-middle tasks: an LLM masks a crucial span of the solution and generates plausible distractors, so correctness is checked by simple option matching, yielding a binary verifiable reward.

Key results

• Adding GooseReason-0.7M to ProRL data lifts Qwen3-4B-Instruct average across math benchmarks from 71.65 to 73.83 (plateaued ProRL data alone 70.36 at +156 steps).
• On cybersecurity, GooseReason-Cyber-4B averages 78.99 vs 67.48 for Llama-Primus-Instruct, a 4.44-point gain over the Qwen3-4B-Instruct base (74.55).
• Continued RL on ProRL-1.5B-v2 with GooseReason yields +2.71% math, +2.12% coding, +3.48% STEM versus +0.63%, +0.95%, +0.13% with ProRL data alone.

Surprising findings

• Open-ended fill-in-the-middle tasks yielded zero accuracy on over 83% of examples for ProRL-1.5B-v2 due to instruction-following failure, so MCQ was used instead.
• Data saturation occurs earlier and is more severe with stronger LLMs; Qwen3-4B-Instruct plateaus or degrades after about 300 RL steps under the ProRL recipe.

Limitations

• Relies on GPT-5 for synthesis, making generation costly and dependent on a closed frontier model.
• Easy FineWeb-derived spans can be inferred from context, requiring difficulty filtering.
• MCQ format may enable elimination strategies on low-distractor settings.

Experimental setup

Datasets: AoPS-Instruct MegaScience rStar-Coder FineWeb ProRL dataset (136K)

Benchmarks: AIME24 AIME25 AMC MATH Minerva OlympiadBench APPS CodeContests CodeForces TACO HumanEvalPlus LiveCodeBench

Metrics: pass@1 average accuracy

Baselines: ProRL-1.5B-v2 RLVE Llama-Primus-Instruct Qwen3-30B-Instruct Llama-3.1-8B-Instruct

Builds on: ProRL GRPO REINFORCE++ rStar-Coder MegaScience

Closest papers by meaning

• Generalization of RLVR Using Causal Reasoning as a Testbed (ICLR 2026 · similarity 0.89)
• Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels (ICLR 2026 · similarity 0.89)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.88)
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.88)
• Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.88)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.88)