Research map / Large language models / Language, safety and interpretability

Confidence Regulation Neurons in Language Models

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Linguistic encoding in LLM representations · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies entropy neurons that regulate output confidence via the final LayerNorm scale by writing to an effective unembedding null space, and discovers token frequency neurons that shift outputs toward or away from the unigram distribution, with an induction case study.

Problem

How large language models internally represent and regulate uncertainty (confidence) in next-token predictions remains largely unexplored; prior neuron findings (entropy neurons) lacked mechanistic explanation and no general-purpose confidence mechanisms were identified.

Approach

Entropy neurons have high weight norm but low logit composition because they write onto an effective null space of the unembedding matrix, so their effect is mediated mainly through the final LayerNorm scale; token frequency neurons instead directly modulate logits along the log-unigram-frequency direction, moving the output toward or away from the unigram distribution.

Key results

• Entropy neurons in GPT-2 Small and LLaMA2 7B have LayerNorm-mediated effects of roughly 80% of total effect, while Pythia and Gemma show around 30-40%.
• Clip mean-ablating GPT-2 Small entropy neuron 11.2378 during induction reduces output entropy by up to 70%.
• Token frequency neurons in Pythia 410M have substantial effects mediated by the token frequency direction, comparable in KL impact to some entropy neurons.

Measured effects reported in the paper

Surprising findings

• The unembedding matrix has an effective null space (sharp singular value drop near index 755 in GPT-2 Small), which entropy neurons write onto.
• Pythia token frequency neuron 23.417 suppresses common tokens and promotes rare ones, pushing outputs toward the frequency distribution in a way opposite to naive expectation.

Limitations

• Focuses only on final-layer entropy and token frequency neurons, not other components.
• Uses proxies (entropy, KL from frequency distribution) rather than a precise definition of confidence.
• Strength of LayerNorm-mediated effect varies across models (about 30-40% in Pythia and Gemma vs about 80% in GPT-2, LLaMA2, Phi-2).

Experimental setup

Datasets: C4 Corpus The Pile OpenWebText

Benchmarks: C4 Corpus

Metrics: total effect direct effect LayerNorm-mediated effect entropy change KL divergence from token frequency distribution loss change reciprocal rank

Baselines: random neurons

Builds on: entropy neurons (Stolfo et al.) direct logit attribution induction heads causal mediation analysis

Closest papers by meaning

• Do LLMs Signal When They’re Right? Evidence from Neuron Agreement (ICML 2026 · similarity 0.86)
• Large language models suffer from a curse of ambiguity (NeurIPS 2026 · similarity 0.86)
• Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.86)
• Integrating Local and Global Entropy for Uncertainty Quantification in LLMs (NeurIPS 2026 · similarity 0.86)
• Confidence-Based Decoding is Provably Efficient for Diffusion Language Models (NeurIPS 2026 · similarity 0.85)
• Distinguishing the Knowable from the Unknowable with Language Models (ICML 2024 · similarity 0.85)