Research map / Large language models / Reasoning and chain of thought

RESTRAIN: From Spurious Votes to Signals — Self-Training RL with Self-Penalization

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RESTRAIN, a label-free GRPO-based RL method that weights all distinct self-consistency pseudo-labels, penalizes low-consensus rollouts, and down-weights low-confidence prompts, nearly matching gold-label GRPO on math/science benchmarks.

Problem

RLVR needs costly gold labels, and label-free methods based on majority voting (e.g., TTRL) suffer from spurious majority votes and training collapse on hard reasoning tasks.

Approach

Replace majority-vote pseudo-labels with frequency-weighted soft pseudo-labels over all distinct model answers, apply zero reward and a negative advantage offset when majority count is low, and scale each prompt's loss by an offline weight from a frozen reference model's self-consistency.

Key results

• On DAPO-14k-MATH with Qwen3-4B-Base, RESTRAIN reaches 51.0 average Pass@1 vs 42.2 for TTRL and 51.4 for gold-label GRPO, surpassing GRPO on MMLU_STEM (80.9 vs 73.7) and GPQA-Diamond (40.2 vs 38.7).
• With OctoThinker Hybrid-8B-Base, RESTRAIN improves AIME25 Pass@1 by 140.7% relative to TTRL (6.5 vs 2.7).
• On test-time RL with Llama3.1-8B-Instruct, RESTRAIN averages 41.4 vs 37.3 for ETMR and 35.3 for TTRL.

Surprising findings

• RESTRAIN without gold labels surpasses gold-label GRPO on MMLU_STEM and GPQA-Diamond, which the authors attribute to reduced overfitting to math-specific patterns.
• Offline-computed prompt weights outperform online-updated prompt weights, which collapse around 100 steps.

Limitations

• Performance is sensitive to hyperparameters (negative offset delta, majority threshold kappa, sigma), which were tuned on the evaluation setting.
• Pass@n degrades when majority size is very low, so rollouts may contain no correct answer and provide noisy signal.
• ETTRL was not reimplemented for large-scale training, so comparisons with it are limited to the test-time setting.

Experimental setup

Datasets: DAPO-14k-MATH Synthetic S1k (CoT-Self-Instruct) AIME24 AMC23 MATH500

Benchmarks: MATH500 AIME25 OlympiadBench Minerva Math MMLU_STEM GPQA-Diamond AMC23 AIME24 S1k

Metrics: Pass@1 averaged over 16 samples average accuracy

Baselines: GRPO (gold labels) TTRL SRT (easy prompt) SRT (offline majority label) ETTRL / ETMR

Builds on: GRPO TTRL Self-Rewarded Training (SRT) Unlikelihood training CoT-Self-Instruct

Closest papers by meaning

• TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning (ICLR 2026 · similarity 0.88)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.88)
• ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
• SELF-HARMONY: LEARNING TO HARMONIZE SELF-SUPERVISION AND SELF-PLAY IN TEST-TIME REINFORCEMENT LEARNING (ICLR 2026 · similarity 0.88)
• Nudging the Boundaries of LLM Reasoning (ICLR 2026 · similarity 0.88)
• LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts (ICLR 2026 · similarity 0.87)