Research map / Large language models / Reasoning and chain of thought

ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressure

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that LRMs produce shorter per-question reasoning when several questions are packed in one prompt ("self-compression"), and proposes ConPress, which SFTs single-question inference on these filtered compressed traces, cutting reasoning tokens 34-49% on math benchmarks.

Problem

Large reasoning models generate long chain-of-thought traces with substantial overthinking, raising inference cost; existing fixes rely on teacher distillation, manual pruning, or RL.

Approach

Multi-question prompts induce per-question reasoning compression (self-compression) even without length instructions; ConPress samples these multi-question generations, keeps only correct per-question traces, and fine-tunes the model on them for single-question prompts, so no external teacher or RL is needed.

Key results

• On Qwen3-4B-Thinking, ConPress cuts average token usage by 48.7% with a 0.6-point average accuracy drop across five benchmarks.
• On R1-Distill-Qwen-7B, ConPress reduces tokens by 33.9% with average accuracy preserved (+0.0), while LC-R1 cuts 45.4% but loses 3.1 points.
• Compared with direct test-time multi-question packing (MQ-N3), ConPress gains +0.9 accuracy while reducing tokens 45.0% on MATH500 and GSM8K for R1-Distill-Qwen-7B.

Surprising findings

• Appending a trivial auxiliary question such as '1+1=?' compresses the target reasoning trace by nearly 50%, with difficulty of the auxiliary question having only modest effect.
• Directly packing test questions at inference (MQ-N3) cuts tokens but drops accuracy by 31.3 points on average, so compression works only when distilled into single-question inference.

Limitations

• Gains on AIME25 are smaller (about 33% token reduction for ConPress on R1-7B, 2-6 points lower compression) and accuracy drops appear on harder benchmarks for Qwen3-4B.
• Evaluation focuses on math benchmarks with 16k/32k generation limits; the out-of-distribution MMLU-STEM test is limited to one comparison.
• Compression-accuracy trade-off is not monotonic in N, and the mechanism behind self-compression is described qualitatively without causal analysis.

Experimental setup

Datasets: MATH AIME (pre-2024) LIMO

Benchmarks: MATH500 AIME25 GSM8K OlympiadBench AMC MMLU-STEM

Metrics: Pass@1 Avg@8 average generated tokens relative token change

Baselines: RFT-shortest DPO-shortest LC-R1 AdaptThink ThinkPrune NoThinking MQ-N2 MQ-N3

Builds on: rejection fine-tuning DPO LIMO

Closest papers by meaning

• Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains (NeurIPS 2025 · similarity 0.89)
• DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains (ICLR 2026 · similarity 0.89)
• When Reasoning Meets Compression: Understanding the Effects of LLMs Compression on Large Reasoning Models (ICLR 2026 · similarity 0.89)
• Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression (ICML 2026 · similarity 0.89)
• Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy (ICLR 2026 · similarity 0.88)
• KaVa: Latent Reasoning via Compressed KV-Cache Distillation (ICLR 2026 · similarity 0.88)