Research map / Large language models / Language, safety and interpretability
Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a neuroscience-inspired neurofeedback paradigm using in-context learning to measure whether LLMs can report and control their own activation projections along target axes (LR and PC), finding reporting and control depend on in-context examples, semantic interpretability, and explained variance.
Problem
Whether LLMs have metacognitive ability to monitor, report, and control their internal activation patterns, which matters for interpretability and for safety against activation-based oversight.
Approach
Labels are generated from the model's own activations projected onto a target axis (logistic-regression or principal-component direction), and the model is given these labels in-context to report them (classification) or to control them (imitation, explicit or implicit), so that success requires monitoring of internal states rather than template matching of training data.
Key results
- Reporting accuracy of Llama-3.1-8B improves with number of in-context examples; LR-axis labels are reported more accurately than PC-axis labels, with earlier PCs reported better.
- Explicit control effects on the LR axis increase with layer depth and model size; Llama-3.1 70B can push activations beyond typical uncontrolled range.
- Implicit control effects are generally similar in direction to explicit ones but much smaller in magnitude, and near zero in early layers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Provide more in-context sentence-label examples | Neurofeedback reporting · ETHICS | reporting accuracy | Helped |
| Provide more in-context sentence-label examples | Neurofeedback reporting · ETHICS | cross-entropy | Helped |
| Target the semantically interpretable LR axis instead of PC axes | Neurofeedback reporting · ETHICS | reporting accuracy | Helped |
| Target earlier PCs rather than later PCs | Neurofeedback reporting · ETHICS | reporting accuracy | Helped |
| Target the LR axis rather than PC axes | Neurofeedback control · ETHICS | Cohen's d | Helped |
| Target earlier PCs rather than later PCs | Neurofeedback control · ETHICS | Cohen's d | Helped |
| Use explicit control instead of implicit control | Neurofeedback control · ETHICS | Cohen's d | Helped |
| Increase the number of in-context examples | Neurofeedback control · ETHICS | Cohen's d | Helped |
| Use deeper layers | Neurofeedback control on the LR axis · ETHICS | Cohen's d | Helped |
| Use a larger model | Neurofeedback control on the LR axis · ETHICS | Cohen's d | Helped |
| Use Llama 3 rather than Qwen 2.5 | Neurofeedback control on the LR axis · ETHICS | Cohen's d | Helped |
Surprising findings
- Llama-3.1 70B can push neural activations along the LR axis beyond the typical range seen in uncontrolled conditions.
- LLM reporting performance is significantly below the ideal observer upper bound, so only a subset of accessible activations are reported.
Limitations
- Control is limited to single-attempt manipulation of a single axis defined at a single layer.
- Off-target effects indicate limited control precision; the controllable space dimensionality is estimated only roughly.
- Focus on residual stream only; other components (attention outputs, MLPs) not analyzed.
Experimental setup
Datasets: ETHICS (commonsense morality) True-False Emotion Sycophancy (synthesized)
Benchmarks: ETHICS True-False Emotion Sycophancy
Metrics: reporting accuracy cross-entropy Cohen's d control effect control precision
Baselines: ideal observer (logistic regression upper bound)
Builds on: in-context learning logistic regression probing principal component analysis logit lens activation patching
Closest papers by meaning
- Tuning-Free Accountable Intervention for LLM Deployment – a Metacognitive Approach (AAAI 2025 · similarity 0.90)
- Decoupling Metacognition from Cognition: A Framework for Quantifying Metacognitive Ability in LLMs (AAAI 2025 · similarity 0.89)
- MP: Endowing Large Language Models with Lateral Thinking (AAAI 2025 · similarity 0.87)
- Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence (ICML 2025 · similarity 0.87)
- Generating Computational Cognitive models using Large Language Models (NeurIPS 2025 · similarity 0.86)
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.