Research map / Large language models / Reasoning and chain of thought
Parameter-Efficient Reinforcement Learning using Prefix Optimization
Itamar Rocha Filho, Rosie Zhao, Sham M. Kakade, Eran Malach, Samy Jelassi
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies whether RLVR gains on math reasoning reflect genuine reasoning improvement or mere steering toward already-present answer formats, using prefix optimization as a parameter-efficient approach.
Problem
RLVR is widely used to tune language models on mathematical reasoning, but it is unclear whether its gains come from real reasoning improvements or from shifting the model toward answer formats already in the reference distribution.
Approach
Investigates the RLVR-gain question through prefix optimization, a parameter-efficient RL method; the abstract is truncated so the precise mechanism is not stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train a 1.5B adapter with RL to generate only the first k tokens | Qwen2.5-7B · MATH-500 | pass@1: 67.4 -> 74.4 | Improved |
| train a 1.5B adapter with RL to generate only the first k tokens | Llama-3.1-70B-Instruct-FP8 · AIME | pass@1: 32.8 -> 49.1 | Improved |
| use a fixed input-independent clustered prefix (k=16) (ablation) | Qwen2.5-7B / Qwen2.5-72B · MATH-500 | pass@1: 67.4 -> 59.4; 82.0 -> 71.8 | Hurt |
| use a fixed input-independent clustered prefix (k=16) (ablation) | Llama-3.1-8B-Instruct · AMC23 | pass@1: 23.4 -> 35.6 | Improved |
| replace Prefix-RL with RL of the full 7B model (ablation) | Qwen2.5-7B · MATH-500 | pass@1: 74 -> 78 | Improved |
| replace Prefix-RL with supervised Prefix-Tuning (ablation) | Qwen2.5-7B · MATH-500 | pass@1: 67.4 -> 45.4 | Hurt |
| replace Prefix-RL with LoRA RL (ablation) | Qwen2.5-7B · AMC23 | pass@1: MATH-500 +2.8, AMC23 -4.0 | Mixed |
| use prefix length k=1 instead of k>=4 (ablation) | Prefix-RL on Qwen2.5-7B · AIME, AMC23, Minerva | pass@1: AIME -1.0, AMC23 -0.9, Minerva -2.2 | Hurt |
| increase prefix length from 32 to 64 (ablation) | Prefix-RL on Qwen2.5-7B · MATH-500 | pass@1: 72.0 -> 72.8 | No clear effect |
| apply MATH-trained prefix adapter to physics (ablation) | Prefix-RL on Qwen2.5-7B · OCW Courses | accuracy: 13.9 -> 20.2 (k=8) | Improved |
| apply MATH-trained prefix adapter to physics with k=64 (ablation) | Prefix-RL on Qwen2.5-7B · UGPhysics | accuracy: 9.0 -> 8.5 | Hurt |
| use Prefix-RL on already strong target | Qwen2.5-72B · AIME | pass@1: 41.2 -> 40.6 | No clear effect |
Limitations
- Only the truncated abstract is available; full method and results are unknown
Experimental setup
Builds on: RLVR
Closest papers by meaning
- Well Begun, Half Done: Reinforcement Learning with Prefix Optimization for LLM Reasoning (AAAI 2026 · similarity 0.93)
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example (NeurIPS 2025 · similarity 0.92)
- The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models (NeurIPS 2026 · similarity 0.92)
- Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs (ICLR 2026 · similarity 0.91)
- From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation (ICLR 2026 · similarity 0.91)
- On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitation (ICLR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
