Research map / Large language models / Reasoning and chain of thought
SemCoT: Accelerating Chain-of-Thought Reasoning through Semantically-Aligned Implicit Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SemCoT, an implicit chain-of-thought method that trains a contrastive sentence transformer to score semantic alignment with ground-truth reasoning and distills reasoning into a lightweight LM's implicit tokens, cutting per-token generation time.
Problem
Implicit CoT methods lose semantic alignment with explicit ground-truth reasoning and still pay high per-token LLM generation cost for each implicit reasoning token.
Approach
A customized sentence transformer built from middle layers of the LLM, trained contrastively on (reasoning, condensed reasoning) pairs, provides a semantic alignment loss that guides a lightweight pruned/distilled LM (Sheared-LLaMA-1.3B) to generate implicit reasoning tokens via a linear projection into the LLM embedding space.
Key results
- On Llama-2-7b-chat, SemCoT reaches 9.83% on GSM8K versus 8.83% for SoftCoT (runner-up) and 1.02s vs 1.05s time, with 46.33% on SVAMP vs 38.17% for SoftCoT.
- On Mistral-7B-Instruct-v0.2, SemCoT reaches 82.17% on CoinFlip versus 66.67% for CODI, while running in about 1.30s.
- SemCoT is not uniformly best: CODI reaches 97.33% on Llama CoinFlip vs SemCoT's 87.00%, and Pause reaches 40.93% on Mistral MultiArith vs SemCoT's 38.89%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SemCoT instead of the strongest baseline | Llama-2-7B-Chat · CoinFlip | accuracy (%): 97.33 -> 87.00 | Hurt |
| Use SemCoT instead of the strongest baseline | Llama-2-7B-Chat · GSM8K | accuracy (%): 8.83 -> 9.83 | Helped |
| Use SemCoT instead of the strongest baseline | Llama-2-7B-Chat · SVAMP | accuracy (%): 38.17 -> 46.33 | Helped |
| Use SemCoT instead of the strongest baseline | Mistral-7B-Instruct-v0.2 · CoinFlip | accuracy (%): 66.67 -> 82.17 | Helped |
| Use SemCoT instead of the strongest baseline | Mistral-7B-Instruct-v0.2 · MultiArith | accuracy (%): 40.93 -> 38.89 | Hurt |
| Use SemCoT instead of the strongest baseline | Mistral-7B-Instruct-v0.2 · SVAMP | accuracy (%): 54.17 -> 53.83 | Hurt |
| Use SemCoT instead of the fastest baseline | Llama-2-7B-Chat · CoinFlip | time (s): 1.08 -> 1.06 | Helped |
| Use SemCoT instead of the fastest baseline | Llama-2-7B-Chat · CommonsenseQA | time (s): 1.01 -> 1.06 | Hurt |
| Use SemCoT instead of the fastest baseline | Mistral-7B-Instruct-v0.2 · GSM8K | time (s): 1.35 -> 1.35 | No clear effect |
| Remove the customized sentence transformer and use cosine similarity of mean-pooled embeddings (ablation) | SemCoT on Llama-2-7B-Chat · reasoning datasets | answer accuracy | Hurt |
| Remove semantic alignment loss from generator training (ablation) | SemCoT on Llama-2-7B-Chat · reasoning datasets | answer accuracy | Hurt |
| Replace the lightweight generator with the original LLM finetuned using LoRA (ablation) | SemCoT on Llama-2-7B-Chat · reasoning datasets | answer accuracy | Hurt |
Surprising findings
- Finetuning the original LLM to generate implicit reasoning (SemCoT-NLL) performed worse than using a lightweight LM, which the authors attribute to catastrophic forgetting.
Limitations
- The customized sentence transformer requires additional training overhead before deployment.
- Evaluation is limited to standard math, commonsense, and symbolic benchmarks and two 7B LLMs.
- Implicit latent reasoning reduces human interpretability.
Experimental setup
Datasets: GSM8K SVAMP MultiArith CommonsenseQA CoinFlip
Benchmarks: GSM8K SVAMP MultiArith CommonsenseQA CoinFlip
Metrics: answer accuracy average wall-clock time
Baselines: Pause ICoT-SI COCONUT CODI SoftCoT
Builds on: SoftCoT COCONUT CODI Sentence-BERT
Closest papers by meaning
- SIM-CoT: Supervised Implicit Chain-of-Thought (ICLR 2026 · similarity 0.92)
- SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.91)
- Eliciting Chain-of-Thought in Base LLMs via Gradient-Based Representation Optimization (AAAI 2026 · similarity 0.90)
- CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs (ICLR 2026 · similarity 0.90)
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.90)
- C3oT: Generating Shorter Chain-of-Thought Without Compromising Effectiveness (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.