Research map / Large language models / Reasoning and chain of thought

Better, Faster: Harnessing Self-Improvement in Large Reasoning Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HSIR, a self-improvement framework for large reasoning models that uses verify-then-exit sampling to recover correct solutions for hard queries and an attention-weighted hidden-state Intrinsic Diversity score to filter overthinking data, plus H-GRPO which adds InDiv as an RL reward.

Problem

Self-improvement training of large reasoning models often underperforms on complex reasoning tasks and can collapse, due to data imbalance (few correct solutions for hard queries) and overthinking (redundant reasoning in self-generated training data).

Approach

Recycle failed solutions by truncating them at the first intermediate step that reaches the ground-truth answer and resampling answers from that prefix (VeriExit), and score reasoning redundancy with an intrinsic, attention-weighted eigenvalue-based diversity measure computed from the model's middle-layer hidden states (InDiv), with InDiv also used as an extra GRPO reward.

Key results

• HSIR-DPO on Qwen2.5 models (MedQA and GSM8K) reaches 73.35% average accuracy after 3 iterations, +10.90 over the initial SFT model, while reducing output tokens by 42.4%.
• HSIR-SFT outperforms ReST-EM on Qwen2.5-1.5B overall average (70.68 vs 69.02 at iteration 2 comparisons; 71.33 vs 69.73 at final iteration), and on LLaMA3-8B gains +1.89 on MedQA over ReST-EM.
• H-GRPO reaches 70.71% overall accuracy vs 69.17% for vanilla GRPO on Qwen2.5 models, with 27% fewer tokens than GRPO.

Surprising findings

• Length-driven filtering (shortest solutions) degrades accuracy, so pure length reduction hurts performance.
• STaR-style self-improvement on MedQA with Qwen2.5-1.5B lowers accuracy from 38.10 to 35.98 after one iteration.

Limitations

• InDiv relies on semantic similarity between adjacent steps, which may misjudge necessary repetitive steps in rigorous math derivations.
• VeriExit cannot help absolutely hard queries where no intermediate step reaches the correct answer.

Experimental setup

Datasets: MedQA GSM8K ARC CommonsenseQA StrategyQA Medbullets MedXpertQA MATH AMC2023 GPQA

Benchmarks: MedQA GSM8K ARC StrategyQA CommonsenseQA Medbullets MedXpertQA MATH AMC2023 GPQA

Metrics: accuracy output tokens

Baselines: STaR ReST-EM RFT ReGenesis IRPO Long2Short CosFn SFT-Oracle

Builds on: STaR ReST-EM DPO GRPO IRPO

Closest papers by meaning

• How Far Are We from Optimal Reasoning Efficiency? (NeurIPS 2025 · similarity 0.89)
• Efficient Reasoning with Balanced Thinking (ICLR 2026 · similarity 0.89)
• Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement (AAAI 2026 · similarity 0.88)
• B-STaR: Monitoring and Balancing Exploration and Exploitation in Self-Taught Reasoners (ICLR 2025 · similarity 0.88)
• Let LRMs Break Free from Overthinking via Self-Braking Tuning (NeurIPS 2025 · similarity 0.88)
• Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.88)