Research map / Large language models / Reasoning and chain of thought
Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Token Signature, using the Spearman correlation between token probabilities and position in greedy standard-prompt decoding to predict CoT gain, and Dynamic CoT, a logistic-regression router choosing CoT or direct answer per question, with voting-based transfer to GPT-4o models.
Problem
CoT prompting improves performance inconsistently across tasks, and there is no reliable measure of when CoT helps; the mechanism is poorly understood and CoT is costly in tokens.
Approach
The monotonicity (Spearman correlation with token index) of the probabilities along the greedy decoding path of a question-only prompt correlates with CoT gain; this signal is used as a benchmark-level indicator (Instance SC, Aggregated SC) and as an instance-level feature for a logistic regression router that selects CoT or direct answer, with open-source voting transferred to closed-source model
Key results
- Aggregated SC predicts CoT gain significance with 89.2% average accuracy across four open-source models, versus 69.6% for Instance SC.
- Dynamic CoT reduces token consumption by 39.1% relative to all-CoT on open-source models while staying near the better of CoT and direct answer.
- Transferring voting-based Dynamic CoT to GPT-4o-mini and GPT-4o reduces token consumption by 35.8% while keeping high accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Aggregated SC rather than Instance SC to predict CoT effectiveness | Token Signature · 12 benchmarks across four open-source models | prediction accuracy: 69.6% -> 89.2% | Helped |
| Select CoT or direct answer using Instance SC and logistic regression | Open-source models · 14 benchmarks | accuracy | Mixed |
| Select CoT or direct answer per instance instead of always using CoT | Open-source models · 14 benchmarks | token consumption: 39.1% reduction | Helped |
| Transfer Dynamic CoT selection by voting across open-source models | Closed-source models · 14 benchmarks | token consumption: 35.8% reduction | Helped |
| Transfer Dynamic CoT selection by voting across open-source models | Closed-source models · GPQA | accuracy: 51.68 -> 47.88 | Hurt |
| Transfer Dynamic CoT selection by voting across open-source models | Closed-source models · StrategyQA | accuracy: 61.77 -> 54.76 | Hurt |
| Use the first 50 tokens for SC prediction rather than other tested token counts (ablation) | Token Signature · CoT effectiveness prediction | prediction accuracy: 69.6% for Instance SC; 89.3% for Aggregated SC | Helped |
| Use few-shot CoT prompting instead of zero-shot CoT prompting (ablation) | Token Signature indicators · CoT effectiveness prediction across four open-source models | prediction accuracy | Mixed |
| Use CoT rather than direct answer | Llama-3.2-3B-Instruct · GSM8K | accuracy: 8.34 -> 72.10 | Helped |
| Use CoT rather than direct answer | Llama-3.2-3B-Instruct · GPQA | accuracy: 37.72 -> 27.01 | Hurt |
| Use CoT rather than direct answer | Llama-3.2-3B-Instruct · MuSR | accuracy: 50.93 -> 44.84 | Hurt |
| Use CoT rather than direct answer | Llama-3.2-3B-Instruct · StrategyQA | accuracy: 79.69 -> 66.59 | Hurt |
Surprising findings
- Instance SC and Aggregated SC often have the opposite sign from CoT gain on some benchmarks, e.g. Mistral-7B on CH_d, indicating the signal is not uniformly reliable.
- Dynamic CoT on GPT-4o-mini and GPT-4o matches CoT exactly on several benchmarks because the voting always selects CoT there.
Limitations
- Benchmark-level prediction accuracy is computed on only 14 model-benchmark pairs per indicator, so estimates are noisy.
- Transfer to closed-source models relies on open-source models' token probabilities via voting and shows a slight drop in classification performance.
- Mechanistic explanation is speculative and based on intuition rather than formal theory.
Experimental setup
Datasets: GSM8K MultiArith FOLIO ContextHub ARC GPQA MuSR AGIEval LSAT CommonsenseQA PIQA SIQA StrategyQA
Benchmarks: GSM8K MultiArith FOLIO ContextHub ARC GPQA MuSR LSAT CommonsenseQA PIQA SIQA StrategyQA
Metrics: accuracy prediction accuracy of indicator token consumption two-tailed Z test significance
Baselines: zero-shot CoT direct answer prompting
Builds on: chain-of-thought prompting zero-shot CoT Spearman correlation
Closest papers by meaning
- On the Token Value Inequality in Efficient Reasoning (NeurIPS 2026 · similarity 0.91)
- Understanding Chain-of-Thought in LLMs through Information Theory (ICML 2025 · similarity 0.91)
- To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (ICLR 2025 · similarity 0.90)
- FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.90)
- Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought (NeurIPS 2024 · similarity 0.90)
- The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.