Research map / Large language models / Reasoning and chain of thought

Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Chain-of-thought training and analysis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LCPO, an offline preference-optimization method trained on about 0.8k self-generated easy-split pairs to shorten reasoning length of R1-Distill-Qwen models by over 50% while roughly preserving math accuracy.

Problem

Large reasoning models produce very long chain-of-thought outputs, raising compute cost and overthinking, while existing efficient-reasoning methods either hurt accuracy or require large-scale online RL.

Approach

Builds preference pairs from the model's own sampled generation space (shortest vs. longest responses on easy questions), then analyzes preference objectives under a unified Bradley-Terry framework and proposes LCPO, which adds a negative counterpart of the NLL-related implicit reward so that the length preference converges without extra hyperparameters.

Key results

• On DeepSeek-R1-Distill-Qwen-7B, LCPO cuts average output length by 54.21% across six math benchmarks with total accuracy change of -1.99 points summed over benchmarks (MATH-500 length 4223 to 2033).
• On DeepSeek-R1-Distill-Qwen-1.5B, LCPO reduces average length by 57.31% with a total accuracy gain of +0.52 across the same six benchmarks.
• LCPO trained on the easy split reaches 92.95% GSM8K accuracy with 52.53% length reduction on the 7B model, outperforming DAST, TrEff and L1 on the length-accuracy trade-off.

Surprising findings

• Training only on the easy split (questions the model already solves) outperforms training on medium or difficult splits in length reduction, and training on difficult splits with mostly incorrect chosen responses still slightly improves accuracy.
• SFT and ORPO reduce length on GSM8K but show accuracy degradation, while the baseline SimPO yields negative length change on GSM8K (-5.96%).

Limitations

• Relies on a coarse pass-rate-based difficulty proxy and correctness labels that the authors admit are heuristic; difficult-split chosen responses are often incorrect.
• Evaluation focuses on math and a few general benchmarks; coding (LiveCodeBench) is listed in appendix but not reported in main results.
• Comparisons with baselines use open-source released checkpoints, and results are on a small set of 1.5B and 7B distilled models.

Experimental setup

Datasets: LIMR MATH-500 GSM8K Minerva-Math AIME24 AMC23 OlympiadBench MMLU GPQA-Diamond WinoGrande

Benchmarks: MATH-500 GSM8K Minerva-Math AIME24 AMC23 OlympiadBench MMLU GPQA-Diamond WinoGrande

Metrics: accuracy (pass@1) average generation length in tokens

Baselines: CoD L1-Exact L1-Max TrEff DAST SFT DPO SimPO ORPO SimPER

Builds on: DPO SimPO ORPO SimPER LIMR DAST

Closest papers by meaning

• Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.91)
• Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMs (NeurIPS 2024 · similarity 0.90)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.90)
• ThoughtFold: Folding Reasoning Chains via Introspective Preference Learning (ICML 2026 · similarity 0.90)
• Demystifying Long Chain-of-Thought Reasoning (ICML 2025 · similarity 0.90)
• OS-Pruner: Pruning Chains-of-Thought of Reasoning Models via Optimal Stopping (NeurIPS 2026 · similarity 0.90)