Research map / Large language models / Reasoning and chain of thought

TraPO: A Semi-Supervised Reinforcement Learning Framework for Boosting LLM Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes semi-supervised RLVR (SS-RLVR) and TraPO, which selects unlabeled math questions whose pass-rate trajectories align with labeled ones and trains GRPO on them with pseudo-labels from majority voting.

Problem

Unsupervised RLVR with self-consistency rewards (majority voting, entropy, self-certainty) can reinforce incorrect consensus and collapse, while fully supervised RLVR needs costly labels. The paper asks how a small labeled set can anchor reward on unlabeled data.

Approach

Use labeled data for ground-truth rewards and select unlabeled samples whose pseudo pass-rate trajectories over training epochs have high cosine similarity to the average trajectory of reliable (labeled) samples; only selected unlabeled samples contribute to the GRPO loss after a warm-up.

Key results

• TraPO with 1K labeled + 3K unlabeled gives 42.6 ID avg and 56.1 OOD avg on Qwen2.5-Math-7B, beating the best unsupervised TTRL-style baseline trained on 45K unlabeled samples by 4.3% ID.
• With 4K labeled + 12K unlabeled, TraPO reaches 45.6 ID and 59.7 OOD average, exceeding fully supervised 45K-label GRPO (45.5 ID, 57.3 OOD).
• Unsupervised training on Llama-3.1-8B-Instruct collapses within tens of steps, while TraPO trains stably.

Surprising findings

• Naive combination of 1K supervised and 3K entropy-based unsupervised RLVR improves only 0.6% accuracy.
• TraPO with 10% labels surpasses fully supervised RLVR trained on all 45K labels on in-distribution average.

Limitations

• Evaluated mainly on math reasoning with Qwen2.5-Math-7B; generalization to other domains such as medicine or finance is asserted but not tested.
• Theoretical bound relies on strong assumptions (NTK regime, gradient alignment) and a sketch proof.
• Pseudo-labels from majority voting may be inaccurate for hard unlabeled samples, which the top-p and threshold selection only partly mitigate.

Experimental setup

Datasets: OpenR1-Math-220k DeepMath

Benchmarks: AIME 2024 AIME 2025 AMC MATH-500 Minerva OlympiadBench ARC-c GPQA-diamond MMLU-Pro

Metrics: avg@32 pass@1 accuracy

Baselines: TTRL Self-certainty Token-level Entropy Sentence-level Entropy Fully supervised GRPO (1K/4K/45K labels)

Builds on: GRPO TTRL DeepSeek-R1 RLVR

Closest papers by meaning

• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.92)
• Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.92)
• On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.92)
• Tricks or Traps? A Deep Dive into RL for LLM Reasoning (ICLR 2026 · similarity 0.92)
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS 2025 · similarity 0.91)
• You Only Need Minimal RLVR Training: Extrapolating LLMs via Rank-1 Trajectories (NeurIPS 2026 · similarity 0.91)