Research map / Large language models / Language, safety and interpretability
Geometry of Decision Making in Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies layer-wise intrinsic dimension (MLE, TwoNN, GRIDE) of last-token representations in 28 open-weight LLMs on MCQA tasks, relating ID peaks to decisiveness measured via logit-lens accuracy.
Problem
How LLMs internally arrive at decisions in multiple-choice QA is opaque; the geometry of hidden representations (intrinsic dimension across layers) and its link to decisiveness and confidence is unexplored.
Approach
Uses intrinsic dimension estimators on last-token residual/MLP activations across layers and pairs them with logit-lens accuracy, finding a hump-shaped ID profile whose peak coincides with the onset of confident predictions, plus analyses of few-shot prompting and Pythia training checkpoints.
Key results
- Across 28 open-weight models, intrinsic dimension of MLP-output last-token representations shows a hump-shaped profile across layers, with ID peaks coinciding with sharp accuracy increases on MMLU.
- Final-layer ID correlates negatively with accuracy for LLaMA models on some datasets (e.g., SST2 Spearman -0.943, Rotten Tomatoes -0.771), but the correlation largely disappears when restricting to models above baseline accuracy.
- Few-shot prompting (up to 50 shots on arithmetic, Pythia 410M-6.9B) generally improves accuracy while reducing final-layer ID, with 2.8B as an exception.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of in-context examples | LLMs on reasoning-heavy MCQA tasks · Reasoning-heavy MCQA tasks | final-layer intrinsic dimension | Helped |
| Increase the number of few-shot examples | Pythia models on arithmetic reasoning · Arithmetic reasoning | accuracy | Mixed |
| Increase the number of few-shot examples | Pythia models on arithmetic reasoning · Arithmetic reasoning | final-layer intrinsic dimension | Helped |
| Use MLP output representations rather than residual post-activations | Transformer layerwise representations · MCQA | intrinsic dimension transition sharpness | Helped |
| Use residual post-activations rather than MLP outputs | Transformer layerwise representations · MCQA | intrinsic dimension trend smoothness | Helped |
| Use final-layer intrinsic dimension as an accuracy proxy | LLaMA-family models · Real-world MCQA datasets | accuracy correlation: -0.720 | Mixed |
| Evaluate only models that perform above baseline | LLaMA-family models · MMLU Social Sciences | Pearson correlation between ID and accuracy: -0.084 -> 0.25 | No clear effect |
| Increase model capacity | Language models · MCQA | layer depth at ID peak | Helped |
Surprising findings
- Final-layer ID is a weak and unstable proxy for model accuracy; correlations vanish for models performing above baseline.
- Smaller models' accuracy sometimes decreases with more few-shot examples, possibly due to surface-pattern extraction.
Limitations
- Correlational rather than causal link between ID peaks and decisiveness.
- ID estimators are noisy; only three estimators used, and only last-token representations considered.
- Restricted to MCQA and two-choice/four-choice formats; open-ended generation not analyzed.
Experimental setup
Datasets: CoLA AG News MMLU Rotten Tomatoes SST2 COPA COLD Arithmetic (template-based) Greater Than (template-based)
Benchmarks: CoLA AG News MMLU Rotten Tomatoes SST2 COPA COLD Arithmetic Greater Than
Metrics: accuracy intrinsic dimension Pearson correlation Spearman correlation Kendall tau
Baselines: MLE TwoNN
Builds on: GRIDE TwoNN Logit Lens TransformerLens
Closest papers by meaning
- Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.88)
- Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions (ICLR 2025 · similarity 0.88)
- Quantifying Generalization Complexity for Large Language Models (ICLR 2025 · similarity 0.88)
- Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.88)
- Tracing the Representation Geometry of Language Models from Pretraining to Post-training (NeurIPS 2025 · similarity 0.87)
- The Tell-Tale Norm: $\ell_2$ Magnitude as a Signal for Reasoning Dynamics in Large Language Models (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.