Research map / Large language models / Reasoning and chain of thought

Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes deep-thinking ratio (DTR), the fraction of tokens whose logit-lens distributions only settle in deep layers, as a measure of reasoning effort that correlates with accuracy, and Think@n, a test-time scaling method that selects samples by DTR from short prefixes to halve self-consistency cost.

Problem

Token counts and confidence measures are unreliable proxies for LLM reasoning effort and quality; length often correlates negatively with accuracy, so a better inference-time measure of thinking effort is needed.

Approach

Project intermediate-layer hidden states to the vocabulary via the unembedding matrix and measure the depth at which each token's distribution settles (JSD to final layer below threshold); tokens settling in the last fraction of layers count as deep-thinking, and DTR is their proportion in a sequence.

Key results

• DTR achieves the highest average Pearson correlation with accuracy (r = 0.687) across eight model variants and four benchmarks, versus -0.594 for token count and 0.605 for Self-Certainty.
• Think@n matches or exceeds Cons@n accuracy on AIME 25 (94.7 vs 92.7 for OSS-120B-medium) at about 49% of the inference cost.
• On OSS-120B-medium, Think@n averages the best accuracy-cost trade-off, with cost reduced roughly 50% relative to Cons@n.

Surprising findings

• Lower GPT-OSS reasoning levels yield higher DTR, while higher reasoning levels give smaller DTR yet better accuracy, so DTR may not be comparable across model modes.
• Token count correlates negatively with accuracy (mean r = -0.59), meaning longer reasoning is generally associated with lower accuracy.

Limitations

• DTR is not directly comparable across models or reasoning modes because it is normalized by sequence length and shifts with reasoning level.
• Hyperparameters (settling threshold g and depth fraction rho) affect the correlation strength, and g=0.25 gives flatter trends.
• Results rely on Pearson correlation over sampled responses across a small set of benchmarks and models.

Experimental setup

Datasets: AIME 2024 AIME 2025 HMMT 2025 GPQA-Diamond

Benchmarks: AIME 2024 AIME 2025 HMMT 2025 GPQA-Diamond

Metrics: Pearson correlation with accuracy accuracy inference cost (tokens)

Baselines: Token count Reverse token count Log probability Negative perplexity Negative entropy Self-Certainty Cons@n Mean@n Long@n Short@n

Builds on: logit lens self-consistency self-certainty early exiting

Closest papers by meaning

• Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.93)
• Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning (NeurIPS 2025 · similarity 0.92)
• Deep Think with Confidence (ICLR 2026 · similarity 0.92)
• Rethinking Thinking Tokens: LLMs as Improvement Operators (ICML 2026 · similarity 0.91)
• Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (ICML 2025 · similarity 0.91)
• Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.91)