Research map / Large language models / Reasoning and chain of thought

Reasoning or Memorization? Unreliable Results of Reinforcement Learning Due to Data Contamination

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows Qwen2.5 math benchmark contamination explains spurious-reward RLVR gains, and proposes RandomCalculation, a leakage-free arithmetic benchmark where only correct rewards help.

Problem

Recent RLVR studies report that random or incorrect rewards improve Qwen2.5 on MATH-500 but not Llama, but it is unclear whether this reflects genuine reasoning or data contamination in widely used math benchmarks.

Approach

Introduces partial-prompt completion and answer-accuracy audits to detect benchmark memorization, plus an automatic generator of arbitrarily long random arithmetic expressions (RandomCalculation) that post-date Qwen's release, with a continuous reward to stabilize RL training.

Key results

• On RandomCalculation, correct reward lets Qwen2.5-Math-7B surpass Max@16 of the initial model, while random and inverted rewards yield no reliable gain or collapse.
• Qwen2.5-Math-7B completes 54.60% of MATH-500 problems exactly from a 60% prefix and answers 53.6% correctly, versus 3.8% and 2.4% for Llama3.1-8B; on LiveMathBench the Qwen completion rate drops to 0.0%.
• Qwen2.5-Math-7B greedy accuracy on MATH-500 without template is 72.20%, and spurious-reward RLVR gains converge to this greedy baseline.

Surprising findings

• Applying the official chat template substantially degrades the Qwen base models' MATH-500 performance.
• Spurious-reward RLVR yields higher ROUGE-L between pre- and post-RL responses on MATH-500 than correct reward, consistent with memory retrieval.

Limitations

• Experiments restricted to a subset of Qwen2.5 and Qwen3 models due to limited compute.
• Did not evaluate many other RL algorithms beyond GRPO.
• RandomCalculation is arithmetic only and may not represent general math reasoning; LiveMathBench RLVR was limited by small training data.

Experimental setup

Datasets: MATH-500 AMC AIME2024 AIME2025 MinervaMath LiveMathBench LiveCodeBench RandomCalculation

Benchmarks: MATH-500 AMC AIME2024 AIME2025 MinervaMath LiveMathBench RandomCalculation

Metrics: accuracy exact match ROUGE-L pass@16 KL divergence partial-prompt completion rate partial-prompt answer accuracy

Baselines: Random reward Inverted reward Mv-incorrect reward Max@16 Greedy (w/o Template)

Builds on: GRPO TTRL RLVR spurious rewards

Closest papers by meaning

• Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.90)
• VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models (ICLR 2026 · similarity 0.89)
• Spurious Rewards: Rethinking Training Signals in RLVR (ICML 2026 · similarity 0.89)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (ICML 2026 · similarity 0.89)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• Reasoning Under 1 Billion: Memory-Augmented Reinforcement Learning for Large Language Models (NeurIPS 2026 · similarity 0.89)