Research map / Large language models / Reasoning and chain of thought
Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that a small set of high-impact MLP activations in the last layers governs long-CoT behavior and proposes EELo-CoT, a training-free activation amplification plus forced "wait" reflection method, along with a 1.51%-parameter fine-tuning variant.
Problem
Eliciting long chain-of-thought reasoning in LLMs normally requires costly RL or SFT on distilled data; the internal mechanisms that give rise to this ability are poorly understood.
Approach
Identify long-CoT-related MLP activations via contrastive positive/negative responses, then amplify them with an analytic decay function (a - b·log(t+c)) triggered after "wait" tokens at inference time, plus a digit-based forced-reflection rule; also a parameter-efficient method that trains a sigmoid-gated amplification module in the last layer with low-rank LoRA elsewhere.
Key results
- Training-free EELo-CoT raises Qwen2.5-7B-base MATH500 accuracy from 69.20% to 72.00% and self-reflection rate from 10.2% to 49.4%, and AMC23 accuracy from 45.00% to 57.50%.
- Parameter-efficient method on Qwen2.5-32B-Instruct trained on LIMO reaches 90.20% on MATH500 and 70.02% on GPQA using 1.51% of parameters, versus LoRA 66.17% on GPQA with 6.15% of parameters.
- EELo-CoT on Qwen2.5-32B-base raises AMC23 accuracy from 45% to 62.5% and reflection rate from 22.5% to 75%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply analytic activation control and forcing reflection | Qwen2.5-7B-base · Math500 | accuracy: 69.20 -> 72.00 | Helped |
| apply analytic activation control and forcing reflection | Qwen2.5-7B-base · AMC23 | accuracy: 45.00 -> 57.50 | Helped |
| apply analytic activation control and forcing reflection | Qwen2.5-7B-base · GPQA | accuracy: 30.30 -> 35.86 | Helped |
| apply analytic activation control and forcing reflection | Qwen2.5-Math-7B-base · Math500 | accuracy: 68.00 -> 76.00 | Helped |
| apply analytic activation control and forcing reflection | Qwen2.5-Math-7B-base · GPQA | accuracy: 33.84 -> 37.88 | Helped |
| use forcing reflection without activation intervention (ablation) | Qwen2.5-7B-base · AMC23 | accuracy: 45.00 -> 40.00 | Hurt |
| use constant activation intervention without forcing reflection (ablation) | Qwen2.5-7B-base · AMC23 | accuracy: 45.00 -> 45.00 | No clear effect |
| combine forcing reflection with constant activation intervention (ablation) | Qwen2.5-7B-base · AMC23 | accuracy: 45.00 -> 47.50 | Helped |
| use forcing reflection instead of the full activation-control rule (ablation) | Qwen2.5-Math-7B-base · Math500 | accuracy: 76.00 -> 64.00 | Hurt |
| use constant activation intervention instead of the full activation-control rule (ablation) | Qwen2.5-Math-7B-base · GPQA | accuracy: 37.88 -> 33.33 | Hurt |
| train with activation amplification and low-rank LoRA | Qwen2.5-32B-Instruct · GPQA | accuracy: 66.17 -> 70.02 | Helped |
| train with activation amplification and low-rank LoRA instead of full fine-tuning | Qwen2.5-32B-Instruct · Math500 | accuracy: 91.60 -> 90.20 | Hurt |
Surprising findings
- Over-post-trained instruct models show near-zero ('dead') activations at the long-CoT-related positions, while base models show the same sparse dynamics as the long-CoT model.
- Amplifying activations alone gives unstable self-reflection; the 'wait' token insertion is needed to induce reliable reflection.
Limitations
- Intervention rules (wait-token insertion, digit-count triggers) are heuristic and empirically chosen.
- Experiments do not cover ultra-large models (72B+) or mixture-of-experts architectures.
- Long CoT may trigger other behaviors (e.g., unverified elaboration) that the method does not disentangle or control.
Experimental setup
Datasets: MATH LIMO
Benchmarks: MATH500 AMC23 GPQA-Diamond
Metrics: accuracy average response length self-reflection rate pass@1
Baselines: Forcing Reflection Constant Intervention Forcing & Constant Full Finetuning LoRA
Builds on: LoRA chain-of-thought prompting vLLM
Closest papers by meaning
- When More is Less: Understanding Chain-of-Thought Length in LLMs (ICLR 2026 · similarity 0.92)
- Demystifying Long Chain-of-Thought Reasoning (ICML 2025 · similarity 0.92)
- Long Chain-of-Thought Reasoning Across Languages (ICLR 2026 · similarity 0.91)
- Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation (NeurIPS 2026 · similarity 0.91)
- Eliciting Chain-of-Thought in Base LLMs via Gradient-Based Representation Optimization (AAAI 2026 · similarity 0.91)
- Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.