Research map / Large language models / Reasoning and chain of thought
Understanding Chain-of-Thought in LLMs through Information Theory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an information-gain metric, estimated from a supervisor model's cross-entropy drop across CoT steps, to detect unidentifiable or incorrect reasoning sub-tasks without step-level annotations.
Problem
Evaluating the correctness of individual intermediate steps in chain-of-thought reasoning without annotated step-level data; existing outcome-based methods (ORM, Math-Shepherd) produce false positives when final-answer correctness is misleading.
Approach
Formalizes CoT as sequential task application in an abstract execution framework and uses conditional mutual information between the final answer and each step (information-gain), estimated via the cross-entropy reduction of a fine-tuned supervisor model, to flag steps that add no information about the correct answer.
Key results
- On the controlled toy setup for LLM_3, IG reaches 0.96 accuracy and 0.06 FPR versus 0.77/0.54 for ORM and 0.60/1.0 for Math-Shepherd.
- On PRM800K sample-wise detection, IG achieves 0.74 accuracy and 0.84 TPR versus 0.69 accuracy and 0.55 TPR for ORM.
- On the Llama-3-8B arithmetic task, IG's information gain drops at step 3 (the failing addition) while ORM's correctness probability stays flat across steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use information-gain detection instead of ORM | Sample-wise step-error detection · Toy Arithmetic | accuracy: 0.56 -> 0.76 | Helped |
| Use information-gain detection instead of Math-Shepherd | Sample-wise step-error detection · Toy Arithmetic | FPR: 0.86 -> 0.02 | Helped |
| Use information-gain detection instead of ORM | Sample-wise step-error detection · Controlled GSM8K | accuracy: 0.58 -> 0.72 | Helped |
| Use information-gain detection instead of ORM | Sample-wise step-error detection · PRM800K | accuracy: 0.69 -> 0.74 | Helped |
| Use information-gain detection instead of ORM | Sample-wise step-error detection · LLM3 sample-wise detection | FPR: 0.54 -> 0.06 | Helped |
| Use information-gain detection instead of Math-Shepherd | Sample-wise step-error detection · LLM3 sample-wise detection | FPR: 1.0 -> 0.06 | Helped |
| Use ORM correctness probability to locate incorrect steps | LLM3 · Toy data | step localization | Hurt |
| Use Math-Shepherd completion proportions to locate incorrect steps | LLM3 · Toy data | step localization | Hurt |
| Use information-gain to assess PRM800K human-labeled steps | CoT step evaluation · PRM800K | information gain | Helped |
| Use information-gain to assess neutral steps | CoT step evaluation · PRM800K | information gain | Hurt |
Surprising findings
- ORM falsely flags the correct 'subtraction' sub-task as incorrect in the controlled GSM8K setting due to spurious correlation with wrong answers.
- Information gain is negative for PRM800K neutral-labelled steps, consistent with those steps adding no useful information.
Limitations
- Requires training a separate supervisor model per setting, which is computationally expensive.
- Still requires labeling each step with its sub-task category.
- Correct final answers with incorrect intermediate steps are not handled; the authors report only 1.2% of arithmetic samples show this.
Experimental setup
Datasets: GSM8K PRM800K Synthetic toy vector-operation data Synthetic arithmetic prompts (x,y in [1,10^5))
Benchmarks: Toy arithmetic GSM8K (controlled) PRM800K
Metrics: accuracy TPR FPR information gain
Baselines: ORM Math-Shepherd
Builds on: Outcome reward model Math-Shepherd Process supervision (PRM800K) LoRA
Closest papers by meaning
- Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought (NeurIPS 2024 · similarity 0.92)
- The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think (ICLR 2026 · similarity 0.92)
- Rethinking Chain-of-Thought from the Perspective of Self-Training (ICML 2025 · similarity 0.91)
- To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (ICLR 2025 · similarity 0.91)
- Deep Hidden Cognition Facilitates Reliable Chain-of-Thought Reasoning (AAAI 2026 · similarity 0.91)
- Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models (ICML 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.