Research map / Large language models / Reasoning and chain of thought
Seq-VCR: Preventing Collapse in Intermediate Transformer Representations for Enhanced Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Seq-VCR, a variance-covariance regularizer on intermediate transformer representations, combined with dummy pause tokens, to prevent representation collapse and solve multi-digit multiplication, arithmetic expression, and LIS tasks without CoT supervision.
Problem
Decoder-only Transformers struggle with multi-step arithmetic reasoning; the paper hypothesizes that representation collapse in intermediate layers limits their ability to compute successive carries and store intermediate results without explicit chain-of-thought supervision.
Approach
Adapts VICReg-style variance and covariance regularization to sequential data (computed per token position across the batch) and applies it to intermediate representations, combined with learnable dummy pause tokens as a cheaper substitute for CoT tokens.
Key results
- On 5x5 multiplication, Seq-VCR + 2 pause tokens reaches 99.5% exact match with fine-tuned GPT-2 Small, versus 0% for vanilla and 44% for GPT-4 5-shot CoT.
- On 4x4 multiplication, Seq-VCR alone raises accuracy from 25% (vanilla) to 52%; with 2 pause tokens it reaches 99.2%.
- Seq-VCR + Pause with 2 pause tokens gives about 5x faster inference than CoT (normalized throughput 0.91-0.95 vs 0.14-0.17), at similar accuracy close to CoT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Seq-VCR + 2 pause tokens instead of vanilla fine-tuning | GPT-2 Small · 4x4 Mult | accuracy (exact match): 0.25 -> 0.992 | Helped |
| Add Seq-VCR + 2 pause tokens instead of vanilla fine-tuning | GPT-2 Small · 5x5 Mult | accuracy (exact match): 0.0 -> 0.995 | Helped |
| Add Seq-VCR to GPT-2 Small fine-tuning | GPT-2 Small · GSM8K | accuracy: 0.191 -> 0.198 | Helped |
| Add Seq-VCR + Pause(2) instead of vanilla fine-tuning | GPT-2 Small · GSM8K | accuracy: 0.191 -> 0.202 | Helped |
| Compute covariance across batch and length dimensions rather than batch only (ablation) | Seq-VCR on GPT-2 Small fine-tuned · 5x5 Mult | accuracy: 0.99 -> 0.98 | No clear effect |
| Apply Seq-VCR at L12 rather than L0 (ablation) | GPT-2 Small fine-tuned with Seq-VCR + Pause · 5x5 Mult | accuracy: 0.97 -> 0.99 | Helped |
| Train from scratch rather than fine-tune, using batch covariance and L12 (ablation) | GPT-2 Small with Seq-VCR · 5x5 Mult | accuracy: 0.99 -> 0.03 | Hurt |
| Train from scratch rather than fine-tune, using batch covariance and L0 (ablation) | GPT-2 Small with Seq-VCR · 5x5 Mult | accuracy: 0.97 -> 0.97 | No clear effect |
| Train from scratch using batch+length covariance rather than batch covariance at L12 (ablation) | GPT-2 Small with Seq-VCR · 5x5 Mult | accuracy: 0.03 -> 0.87 | Helped |
| Add Seq-VCR during fine-tuning | Llama3.2-1B with LoRA(r=64) · 5x5 Mult | accuracy: 0 -> 0.974 | Helped |
| Vary the number of pause tokens in Seq-VCR + Pause (ablation) | 5-layer model on Arithmetic Expressions · Arithmetic Expressions, 4–6 operators | accuracy | No clear effect |
| Add pause tokens to Vanilla on the low-complexity four-operator task (ablation) | 5-layer Vanilla model · Arithmetic Expressions, 4 operators | accuracy | Hurt |
Surprising findings
- Pause tokens alone did not help GPT-2 Small on 5x5 multiplication (0% accuracy), and in the arithmetic expression ablation pause tokens reduced vanilla accuracy on the easiest setting.
- Training GPT-2 from scratch is more sensitive to hyperparameters; applying Seq-VCR at the last layer performed poorly from scratch, while regularizing the first layer was effective.
Limitations
- Results on 5x5 multiplication are from a single fine-tuned GPT-2 Small with heavy hyperparameter tuning done manually; the paper reports 3 seed runs only for some figures.
- The GSM8K gains are marginal (Vanilla 0.191 vs Seq-VCR + Pause 0.202) and far below CoT (0.437).
- Pause token count showed no clear correlation with task complexity, and the authors speculate this stems from shared embeddings across pause tokens.
Experimental setup
Datasets: BIG-bench multi-digit multiplication (4x4, 5x5) Arithmetic Expression dataset Longest Increasing Subsequence (LIS) dataset GSM8K (enhanced version) C4 CodeXGLUE text-to-code
Benchmarks: 4x4 multiplication 5x5 multiplication Arithmetic Expression LIS GSM8K CodeXGLUE text-to-code
Metrics: exact match accuracy matrix-based entropy validation perplexity normalized throughput
Baselines: Vanilla fine-tuning Pause tokens With CoT GPT-4 5-shot CoT GPT-3.5 5-shot Seq-VCR
Builds on: VICReg VCReg pause training matrix-based Renyi entropy
Closest papers by meaning
- Chain of Thought Empowers Transformers to Solve Inherently Serial Problems (ICLR 2024 · similarity 0.88)
- Transformers Can Do Arithmetic with the Right Embeddings (NeurIPS 2024 · similarity 0.87)
- Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers (NeurIPS 2025 · similarity 0.86)
- Transformers Provably Solve Parity Efficiently with Chain of Thought (ICLR 2025 · similarity 0.86)
- Distilling Sequential Computation in Transformer Language Models (NeurIPS 2026 · similarity 0.86)
- Grokking of Implicit Reasoning in Transformers: A Mechanistic Journey to the Edge of Generalization (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.