AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Parameter-Efficient Reinforcement Learning using Prefix Optimization

Itamar Rocha Filho, Rosie Zhao, Sham M. Kakade, Eran Malach, Samy Jelassi

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Studies whether RLVR gains on math reasoning reflect genuine reasoning improvement or mere steering toward already-present answer formats, using prefix optimization as a parameter-efficient approach.

Problem

RLVR is widely used to tune language models on mathematical reasoning, but it is unclear whether its gains come from real reasoning improvements or from shifting the model toward answer formats already in the reference distribution.

Approach

Investigates the RLVR-gain question through prefix optimization, a parameter-efficient RL method; the abstract is truncated so the precise mechanism is not stated.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train a 1.5B adapter with RL to generate only the first k tokensQwen2.5-7B · MATH-500pass@1: 67.4 -> 74.4Improved
train a 1.5B adapter with RL to generate only the first k tokensLlama-3.1-70B-Instruct-FP8 · AIMEpass@1: 32.8 -> 49.1Improved
use a fixed input-independent clustered prefix (k=16) (ablation)Qwen2.5-7B / Qwen2.5-72B · MATH-500pass@1: 67.4 -> 59.4; 82.0 -> 71.8Hurt
use a fixed input-independent clustered prefix (k=16) (ablation)Llama-3.1-8B-Instruct · AMC23pass@1: 23.4 -> 35.6Improved
replace Prefix-RL with RL of the full 7B model (ablation)Qwen2.5-7B · MATH-500pass@1: 74 -> 78Improved
replace Prefix-RL with supervised Prefix-Tuning (ablation)Qwen2.5-7B · MATH-500pass@1: 67.4 -> 45.4Hurt
replace Prefix-RL with LoRA RL (ablation)Qwen2.5-7B · AMC23pass@1: MATH-500 +2.8, AMC23 -4.0Mixed
use prefix length k=1 instead of k>=4 (ablation)Prefix-RL on Qwen2.5-7B · AIME, AMC23, Minervapass@1: AIME -1.0, AMC23 -0.9, Minerva -2.2Hurt
increase prefix length from 32 to 64 (ablation)Prefix-RL on Qwen2.5-7B · MATH-500pass@1: 72.0 -> 72.8No clear effect
apply MATH-trained prefix adapter to physics (ablation)Prefix-RL on Qwen2.5-7B · OCW Coursesaccuracy: 13.9 -> 20.2 (k=8)Improved
apply MATH-trained prefix adapter to physics with k=64 (ablation)Prefix-RL on Qwen2.5-7B · UGPhysicsaccuracy: 9.0 -> 8.5Hurt
use Prefix-RL on already strong targetQwen2.5-72B · AIMEpass@1: 41.2 -> 40.6No clear effect

Limitations

Experimental setup

Builds on: RLVR

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy