Research map / Architectures and efficiency / Attention and transformers

Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Catch Your Breath (CYB), a sequential-decision loss that lets a transformer emit a don't-know token to self-determine how many pause steps to use per input token, outperforming Think Before You Speak cross-entropy pause training at matched data, compute and memory.

Problem

Existing width-based inference-time scaling with pause tokens trains models to read out only at the final pause using cross-entropy, so the model cannot regulate its own processing or signal readiness to respond.

Approach

Frame per-token output as a sequential decision: at each pause step the model emits a don't-know (abstention) probability and the loss is the negative log of the expected discounted accuracy over the self-determined readout step, letting the model choose its own latency with parallel training.

Key results

• CYB with 3 pauses reduces C4 perplexity by +2.72% relative to TBYS with 3 pauses on fine-tuned Gemma1-2B (11.657 vs 11.983), and +3.57% in pretraining from scratch.
• On Gemma3-4B fine-tuning, 1-pause CYB already beats 3-pause TBYS; 3-pause CYB gives +2.20% relative perplexity improvement.
• Downstream top-1 accuracy with Gemma3-4B: CYB 27.94 HellaSwag, 54.00 MMLU, 53.67 GSM8K versus TBYS 24.74, 52.70, 53.37 and baseline 24.68, 49.16, 47.60.

Surprising findings

• Even without early-stopping pressure or discounting, the CYB model learns to pause selectively rather than always using maximum steps.
• Forcing early stopping via external stop-time distributions shifted the learned latency distribution to longer delays and severely hurt perplexity.

Limitations

• Training inserts the maximum number of pauses after every token, which consumes context window and limits effective length for long-form generation.
• Gains on downstream tasks are modest and GSM8K required extra fine-tuning since zero-shot accuracy was 0%.
• Experiments limited to Gemma1-2B and Gemma3-4B on C4 with a single pause-insertion recipe.

Experimental setup

Datasets: C4 GSM8K

Benchmarks: C4 validation perplexity HellaSwag MMLU GSM8K

Metrics: perplexity top-1 accuracy relative perplexity improvement

Baselines: Think Before You Speak (TBYS) baseline without pauses baseline 4x seq CYB-VA CYB-DP

Builds on: Think Before You Speak RoPE adaptive computation time

Closest papers by meaning

• EAGer: Entropy-Aware GEneRation for Adaptive Inference-Time Scaling (ICML 2026 · similarity 0.88)
• ATTS: Asynchronous Test-Time Scaling via Conformal Prediction (ICLR 2026 · similarity 0.87)
• Reinforced Fast Weights via Next-Sequence Prediction (NeurIPS 2026 · similarity 0.87)
• Predicting LLM Output Length via Entropy-Guided Representations (ICLR 2026 · similarity 0.87)
• Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.86)
• Kinetics: Rethinking Test-Time Scaling Law (NeurIPS 2025 · similarity 0.86)