Research map / Reinforcement learning / RL from feedback
Hide to Guide: Learning via Semantic Masking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SMEPO, which masks reward-relevant spans (numbers, code blocks, answer entities) in expert traces while keeping their procedural structure, and uses the masked traces as guidance during GRPO-style RLVR across math, code, and agentic search.
Problem
Expert traces help RLVR exploration on hard problems, but unmodified traces expose verifier-relevant content (final answers, intermediate values, code, entities) that the policy can copy as a reward hack instead of learning the underlying reasoning.
Approach
Instead of truncating or revealing expert traces wholesale, mask only the reward-relevant semantic spans on the critical path (numeric values, executable code blocks, answer entities) while preserving decomposition, plan, and procedural structure, turning guidance into fill-in-the-blank exploration hints without changing the reward or RL objective.
Key results
- On Qwen3-8B-Base math, SMEPO reaches a six-benchmark average of 50.3 vs 47.6 for GRPO and 41.0 for full expert-trace conditioning.
- On Bamboogle with Qwen3-8B-Base, SMEPO scores 48.8 vs 45.6 for GRPO and 23.2 for full expert traces.
- SMEPO reaches the GRPO hard-math target faster, e.g. 2.37 h vs 9.90 h on Qwen3-8B-Base, and reports up to 4.2x training-time reduction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| append full expert traces during GRPO (vs. vanilla GRPO) | Qwen3-8B-Base math with DeepSeek-R1 or Gemini traces · Math 1k, six benchmarks | avg accuracy: -6.6 (DS), -6.8 (Gemini) | Hurt |
| mask numeric spans in expert trace (SMEPO) vs. vanilla GRPO | Qwen3-8B-Base math · Math 1k, six benchmarks | avg accuracy: +2.7 | Helped |
| mask numeric spans in expert trace vs. full expert trace | Qwen3-8B-Base math · Math 1k, six benchmarks | avg accuracy: +9.3 to +9.5 | Helped |
| full expert traces appended during GRPO | Qwen2.5-7B math · Math 1k, six benchmarks | avg accuracy: -3.8 | Hurt |
| mask numeric spans in expert trace vs. vanilla GRPO | Qwen2.5-7B math · Math 1k, six benchmarks | avg accuracy: +2.1 (42.4 to 44.5) | Helped |
| full expert traces appended during GRPO | DeepSeek-R1-Distill-Qwen-7B math · Math 1k, six benchmarks | avg accuracy: -1.4 | Hurt |
| mask numeric spans in expert trace vs. vanilla GRPO | DeepSeek-R1-Distill-Qwen-7B math · Math 1k, six benchmarks | avg accuracy: +0.8 (61.8 to 62.6) | Helped |
| append full expert code trace during GRPO | Qwen3-8B-Base code · Code 3-benchmark avg | pass@1 avg: -0.8 | Hurt |
| mask expert code blocks (SMEPO) vs. vanilla GRPO | Qwen3-8B-Base code · Code 3-benchmark avg | pass@1 avg: +2.5 | Helped |
| append full expert search trace during GRPO | Qwen3-8B-Base agentic search · Bamboogle | accuracy: -22.4 | Hurt |
| mask answer-relevant entities in expert search trace vs. vanilla GRPO | Qwen3-8B-Base agentic search · Bamboogle | accuracy: +3.2 | Helped |
| add SMEPO masking to GHPO | Qwen3-8B-Base GHPO · Math 1k, six benchmarks | avg accuracy: +1.6 (44.0 to 45.6) | Helped |
Surprising findings
- Full expert-trace conditioning underperforms vanilla GRPO across all reported settings, even with strong teacher traces from DeepSeek-R1 or Gemini-3-Pro.
- Random word or sentence masking at matched budget performs worse than full expert traces in math, and non-monotonic prefix exposure remains below SMEPO.
Limitations
- Masking rules are domain-specific heuristics (number matching, code-block detection, spaCy NER) and must be designed per domain.
- Evaluated only on three open-weight models of 7-8B size and on small 1k-example training subsets for the main results.
Experimental setup
Datasets: OpenThoughts-114k MiroVerse HotpotQA search trajectories GHPO data
Benchmarks: GSM8K MATH500 AIME25 AIME26 AMC OlympiadBench HumanEval HumanEval+ LiveCodeBench Bamboogle MMLU
Metrics: accuracy pass@1 mean@32
Baselines: GRPO SFT SFT + RL Expert (full trace) LUFFY GHPO Expert prefix exposure random word masking random sentence masking
Builds on: GRPO LUFFY GHPO
Closest papers by meaning
- The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.89)
- Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs (ICLR 2026 · similarity 0.89)
- From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation (ICLR 2026 · similarity 0.89)
- Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward (ICLR 2026 · similarity 0.88)
- LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards (ICLR 2026 · similarity 0.88)
- DeepSearch: Overcome the Bottleneck of Reinforcement Learning with Verifiable Rewards via Monte Carlo Tree Search (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.