Research map / Large language models / Reasoning and chain of thought

REA-RL: Reflection-Aware Online Reinforcement Learning for Efficient Reasoning

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes REA-RL, an online RL framework for efficient reasoning that adds a 7B reflection model for sequential revision of overthinking traces and a reflection reward penalizing low-reflection-density responses, cutting token cost by about 36% without accuracy loss on math benchmarks.

Problem

Large reasoning models overthink, incurring high inference cost; online RL with length rewards shortens responses but tends to remove reflection, hurting accuracy on hard problems.

Approach

Uses a small distilled reflection model (fine-tuned Qwen2.5-7B-Instruct) to truncate overthinking tokens after the first correct answer and regenerate a short revision online, adding these revised paths to GRPO training, plus a reflection-density reward that penalizes only the lowest-20% reflection-density responses to counter the length reward's bias against reflection.

Key results

• REA-RL (RLen+Reflect combined with reflection model) reduces average token cost to 63.51% at 16k budget with 80.74% accuracy versus 80.39% original, and the reflection-reward-only variant reaches 82.13% accuracy at 71.77% token ratio.
• Plain length reward GRPO R_Len drops 16k average accuracy to 76.88% while cutting tokens to 57.23%, showing performance loss from length reward alone.
• The 7B reflection model achieves 80.62% accuracy at 83.50% token ratio with one-step revision, comparable to the 32B two-step revision (80.61%, 75.65%).

Surprising findings

• Length reward alone causes severe accuracy loss, and the reflection model's gains come with reduced compression under the 8k budget, but accuracy gains can appear after revision because truncation sometimes lets the model state a correct answer that it failed to output within budget.

Limitations

• Validated only on distilled 7B and 1.5B LRMs, not on models pretrained from scratch.
• The LLM-based overthinking detection cannot guarantee complete removal of overthinking.
• Sequential revision adds about 10% cost versus parallel sampling alone due to poorer vLLM parallelism.

Experimental setup

Datasets: DeepScaleR MMLU MMLU-Pro SuperGPQA

Benchmarks: GSM8K MATH500 Gaokao23 AMC23 AIME24 MMLU MMLU-Pro SuperGPQA

Metrics: accuracy token ratio reflection density

Baselines: GRPO GRPO R_Len (Kimi K1.5 length reward) SFT RPO NoThink ShorterBetter Light-R1 DAST

Builds on: GRPO Kimi K1.5 length reward RPO LoRA

Closest papers by meaning

• Stop Unnecessary Reflection: Training LRMs for Efficient Reasoning with Adaptive Reflection and Length Coordinated Penalty (ICLR 2026 · similarity 0.91)
• Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement (ICML 2026 · similarity 0.90)
• Learn to Reason Efficiently with Adaptive Length-based Reward Shaping (ICLR 2026 · similarity 0.89)
• SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning (NeurIPS 2025 · similarity 0.89)