Research map / Large language models / Reasoning and chain of thought

HAPO: Training Language Models to Reason Concisely via History-Aware Policy Optimization

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes HAPO, a GRPO-based RL method whose length reward is computed relative to a per-problem history state (minimum length of past correct responses), reducing response length 33-59% with 2-5% accuracy drops on math benchmarks.

Problem

Reasoning LLMs produce verbose outputs at test time, and existing efficiency methods use universal budgets or in-batch/query-level length comparisons that ignore history from earlier encounters with the same problem during training.

Approach

Track a per-query history reference length (minimum length of previously correct responses) and reward correct responses shorter than it, with a cosine-shaped length reward that does not overly penalize short incorrect responses, combined with a binary correctness reward.

Key results

• On DeepSeek-R1-1.5B, HAPO averages 49% length reduction (7710 to 3937 tokens) with Pass@1 dropping from 64.06 to 62.23 across GSM8K, MATH500, and AIME2024.
• On DeepScaleR-1.5B, HAPO reduces average tokens from 4764 to 3205 (33%) with about 5% accuracy drop (70.84 to 66.31 average Pass@1 across three benchmarks); on Qwen-2.5-1.5B-Inst, tokens fall 59% (1681 to 694) with Pass@1 dropping 42.49 to 40.99.
• Against baselines on DeepSeek-R1-1.5B, HAPO uses 19% fewer tokens than the strongest baseline on AIME2024 and 23% fewer on average than L1-Exact while keeping similar accuracy.

Surprising findings

• Training with correctness reward only (w=0) also reduces response length substantially, which the authors attribute possibly to low difficulty of the training questions.
• Using the mean instead of the minimum for the history state yields even shorter responses but lower accuracy.

Limitations

• Evaluated only on math problems for training and primarily math evaluation; out-of-domain gains on GPQA are mixed and smaller.
• Uses a single scalar history reference (minimum); richer history such as length distributions is not explored.
• Accuracy drops of 2-5% and gains are less pronounced on already-optimized DeepScaleR-1.5B.

Experimental setup

Datasets: DeepScaleR-Preview-Dataset MATH train split GSM8K MATH500 AIME2024 GPQA LiveCodeBench

Benchmarks: GSM8K MATH500 AIME2024 GPQA LiveCodeBench

Metrics: Pass@1 #Tokens

Baselines: L1-Exact L1-Max Query-Opt O1-Pruner HAPO w/ w=0

Builds on: GRPO PPO L1 O1-Pruner DeepScaleR

Closest papers by meaning

• Thinking on the Fly: Test-Time Reasoning Enhancement via Latent Thought Policy Optimization (ICLR 2026 · similarity 0.90)
• Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space (NeurIPS 2026 · similarity 0.90)
• LoongRL: Reinforcement Learning for Advanced Reasoning over Long Contexts (ICLR 2026 · similarity 0.90)
• Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.89)
• IAPO: Information-Aware Policy Optimization for Token-Efficient Reasoning (ICML 2026 · similarity 0.89)
• Probing to Refine: Reinforcement Distillation of LLM Reasoners via Explanatory Inversion (ICLR 2026 · similarity 0.89)