Research map / Large language models / Reasoning and chain of thought

Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes self-rewriting, where an LRM rewrites its own correct reasoning on "simple" queries during GRPO training and learns from the rewrites, improving accuracy-length tradeoff and internal reasoning quality.

Problem

Outcome-only correctness rewards in RL for large reasoning models leave internal reasoning flaws (over-, under-, redundant-, disordered-thinking) unaddressed, and length-control methods ignore these quality issues.

Approach

Selectively rewrites reasoning traces only for queries where all initial rollouts are correct, assigns reward 1 to correct rewritten samples (0 to other correct ones), and runs rewriting and vanilla generation in one batch within GRPO to keep overhead around 10%.

Key results

• On Qwen3-8B, self-rewriting reaches 78.8 average accuracy (+0.6 over original) with 46% shorter reasoning (2293 vs 4243 tokens).
• Self-rewriting achieves a judge score of 79.3 (+7.2 over original) on Qwen3-8B, the highest among compared methods.
• Rewriting adds about 10% time overhead relative to vanilla GRPO according to the authors.

Surprising findings

• Rewriting reduces length even without explicit length-reduction instructions in the rewriting prompt.
• Direct rewriting yields larger LLM-judge score gains than training with the self-rewriting framework, which the authors attribute to correctness rewards preventing overfitting to rewrites.

Limitations

• Evaluation of reasoning quality relies on LLM judges rather than human assessment.
• Only the minimal general rewriting prompt is tested; targeted rewriting is left for future work.
• Experiments limited to Qwen3 models of 1.7B to 8B and a 10K-sample training set.

Experimental setup

Datasets: DeepMath-103K MMLU

Benchmarks: MATH-500 GPQA-Diamond ARC-Challenge MMLU-Pro

Metrics: accuracy (pass@1) reasoning length (tokens) LLM-as-a-judge score

Baselines: GRPO LenPen1 LenPen2 ShorterBetter LPO TOPS

Builds on: GRPO SimPO ShorterBetter LPO TOPS

Closest papers by meaning

• SRPO: Self-Reflective Policy Optimization for Long-Horizon Reasoning (ICML 2026 · similarity 0.90)
• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.89)
• Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards (NeurIPS 2025 · similarity 0.89)
• Incentivizing LLMs to Self-Verify Their Answers (NeurIPS 2025 · similarity 0.89)
• SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Models (ICLR 2026 · similarity 0.89)
• Self-Verifying Reflection Helps Transformers with CoT Reasoning (NeurIPS 2025 · similarity 0.89)