Research map / Large language models / Reasoning and chain of thought

ThoughtFold: Folding Reasoning Chains via Introspective Preference Learning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: RL with verifiable rewards for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ThoughtFold, which uses introspective prune-and-verify search to find redundant steps in correct reasoning traces and trains with a step-masked DPO objective jointly with GRPO to shorten chains of thought while keeping accuracy.

Problem

Outcome-based RLVR reinforces redundant exploration inside correct long chain-of-thought trajectories, causing overthinking in large reasoning models.

Approach

Identify redundant reasoning steps in outcome-correct trajectories via prune-and-verify search (tail truncation then attention-guided internal folding), build preference pairs between shorter and longer valid chains, and apply a dynamic step-level mask to a DPO objective (fold-anchor based) added to GRPO.

Key results

• On DeepSeek-R1-Distill-Qwen-7B, ThoughtFold reduces overall tokens by 56.1% (10,234 to 4,496) while overall accuracy rises from 64.56 to 67.38.
• On Qwen3-8B, ThoughtFold reaches 79.00 overall accuracy vs 77.08 for GRPO, with 5,874 tokens vs 9,932.
• Removing the dynamic mask drops Qwen3-8B overall accuracy to 75.80, below the GRPO baseline of 77.08.

Surprising findings

• Without step-level masking, the method's accuracy falls below plain GRPO, which the authors attribute to credit assignment ambiguity.
• Short-RL and length-penalty baselines lose accuracy overall while ThoughtFold gains accuracy alongside length reduction.

Limitations

• S-GRPO results are official numbers reported by the original work because code was unavailable, so comparisons are not fully controlled.
• Only one training dataset (DeepMath-103K, math) is used; out-of-domain generalization is evaluated only on GPQA.

Experimental setup

Datasets: DeepMath-103K GSM8K MATH-500 AIME 2024 AIME 2025 GPQA Diamond

Benchmarks: GSM8K AIME 2024 AIME 2025 MATH-500 GPQA Diamond

Metrics: accuracy (pass@1) token count

Baselines: GRPO RL + Length Penalty Short-RL S-GRPO

Builds on: GRPO DPO Mask-DPO RLVR S-GRPO

Closest papers by meaning

• Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization (ICLR 2026 · similarity 0.90)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.90)
• Don’t Think Longer, Think Wisely: Optimizing Thinking Dynamics for Large Reasoning Models (NeurIPS 2025 · similarity 0.89)
• Demystifying Long Chain-of-Thought Reasoning (ICML 2025 · similarity 0.89)
• DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains (ICLR 2026 · similarity 0.89)
• Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy (ICLR 2026 · similarity 0.89)