Research map / Large language models / Reasoning and chain of thought
Do LLMs Signal When They’re Right? Evidence from Neuron Agreement
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Neuron Agreement Decoding (NAD), an unsupervised best-of-N method that selects reasoning samples using neuron activation sparsity and cross-sample neuron agreement, enabling early correctness prediction and token-saving early stopping.
Problem
Label-free best-of-N selection for LLM reasoning relies on external signals (token probabilities, entropy, self-evaluation) that can be poorly calibrated after post-training, and majority voting is inapplicable to open-ended outputs such as code.
Approach
Uses internal neuron activation sets rather than output-level signals: correct responses activate fewer unique neurons and show higher cross-sample neuron agreement (Jaccard similarity), so selection by kNN agreement, medoid, DBSCAN clustering, or minimum activation can be done early (first 32 tokens) to prune traces.
Key results
- On AIME24+25 with Qwen3-4B-Thinking, NAD-kNN reaches 85.0 vs 86.7 for majority voting (Cons@64) and 74.6 for Avg@64.
- Early stopping after the first 32-token chunk cuts total tokens by about 97-99% on scientific benchmarks; NAD-MinAct reaches 62.4 average accuracy on Qwen3-4B-Instruct vs 55.5 for Avg@64.
- On code benchmarks, NAD-kNN averages 79.9 vs 76.7 for Avg@64 on Qwen3-4B-Thinking across HumanEval, LCBv5, and MBPP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select candidate by kNN neuron agreement instead of Avg@64 random sampling | Qwen3-4B-Think · AIME24+25, GPQA, HumanEval, LCBv5, MBPP | average accuracy: 79.9 vs 76.7 average | Helped |
| NAD-kNN selection matches majority voting on math/science | Qwen3-4B-Think · AIME24+25 | accuracy: 85.0 vs 86.7 Cons@64 | No clear effect |
| select candidate with fewest activated neurons (MinAct) | Qwen3-4B-Instruct · AIME24+25 | accuracy: 61.7 vs 51.7 Avg@64 | Helped |
| NAD selection on coding (LCBv5) with Qwen3-4B-Think | Qwen3-4B-Think · LCBv5 | accuracy: 61.7 vs 61.9 Avg@64 | No clear effect |
| early stopping after first 32-token chunk with neuron-based selection | Qwen3-4B-Think · AIME24+25 early stop | tokens (M): 1.3M vs 55.2M, -97.6% | Helped |
| minimal-activation selection vs maximal-activation selection (ablation) | Qwen3-4B (math/science) · AIME24 reasoning benchmarks | accuracy: selected-min significantly higher than selected-max (text only) | Helped |
| minimal-activation selection vs maximal-activation selection (ablation) | code generation · code benchmarks | accuracy | Mixed |
| top-k neuron filtering across sequences (k=2K to 200K) vs union (no top-k) (ablation) | activated neuron set aggregation · AIME24 neuron distribution analysis | separation of correct vs incorrect activation distributions | Hurt |
| stopping at 4096 tokens instead of full response for kNN (ablation) | NAD-kNN on Qwen3-4B-Think · AIME24+25 | accuracy: 86.7 vs 85.0 | Mixed |
| early stopping at 32 tokens with NAD selection | Qwen3-4B-Think · GPQA | accuracy: 68.2 vs 66.3 Avg@64 (Medoid) | Helped |
Surprising findings
- On code generation, selecting maximal-activation trajectories sometimes outperforms minimal-activation ones, weakening the activation-correctness link.
- Stopping later (e.g., at 4096 tokens) can give higher accuracy than using the full response (86.7 vs 85.0 on AIME24+25), suggesting later tokens add noise.
Limitations
- Storing neuron activations for all samples requires substantial disk and memory.
- Did not determine which selector is most effective under different sampling patterns.
- Neuron-correctness relationship is weaker on open-ended code generation.
Experimental setup
Datasets: AIME24 AIME25 GPQA LiveCodeBench v5 HumanEval MBPP
Benchmarks: AIME24 AIME25 GPQA HumanEval LiveCodeBench v5 MBPP
Metrics: accuracy Avg@64 Cons@64 total token consumption
Baselines: Avg@64 Cons@64 (majority voting) NAD-MinAct (ablation of NAD variants)
Builds on: self-consistency DeepConf self-certainty model utility law
Closest papers by meaning
- Towards Neuron Attributions in Multi-Modal Large Language Models (NeurIPS 2024 · similarity 0.87)
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.87)
- LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations (ICLR 2025 · similarity 0.87)
- Benford’s Curse: Tracing Digit Bias to Numerical Hallucination in LLMs (NeurIPS 2025 · similarity 0.86)
- Confidence Regulation Neurons in Language Models (NeurIPS 2024 · similarity 0.86)
- Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.