Research map / Architectures and efficiency / RNNs and normalization
What does the Knowledge Neuron Thesis Have to do with Knowledge?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Re-evaluates the Knowledge Neuron thesis by localising BLiMP syntactic phenomena with integrated-gradient KN search, testing KN edit and ROME under new symmetry and synonymy criteria, and concluding MLP neurons store token expression patterns rather than knowledge.
Problem
Whether the Knowledge Neuron thesis, that factual knowledge is stored in MLP neurons as key-value memories editable by neuron or weight edits, actually holds as an account of how LMs recall facts and express syntax.
Approach
Applies the KN-edit localisation pipeline to syntactic minimal-pair phenomena (BLiMP) to show syntax localises to a few neurons like facts, introduces symmetry and synonymy criteria for editing evaluation, and proposes a subspace similarity measure (R_1^2) for localisation level.
Key results
- Determiner-number agreement localises to two neurons: a plural neuron (w^9_1094) present in 100% of plural Det-N pairs and a singular neuron (w^10_2096) in 93% of 'this' pairs.
- KN edit overturns the categorical prediction for at most 5.2% of BLiMP results, and reliability on facts ranges from 1.66% to 47.86%.
- ROME reaches 96-100% reliability on ParaRel bijective and synonym edits but only 23.71%-33.64% symmetry and 52.35%-58.36% synonymy under the new criteria.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Suppress the plural number neuron w^9_1094 (ablation) | BERT KN edit · BLiMP determiner_noun_agreement_2 | probability of plural modifiers | Hurt |
| Suppress the singular number neuron w^10_2096 (ablation) | BERT KN edit · BLiMP determiner_noun_agreement_2 | probability of singular modifiers | Mixed |
| Use KN edit to alter the target association | KN edit · BLiMP syntactic phenomena | prediction overturn rate: at most 5.2% | Hurt |
| Apply ROME edits under synonymy evaluation | ROME · P101 synonym evaluation | synonym: 52.35%; 58.36% | Mixed |
| Apply ROME edits under bijective symmetry evaluation | ROME · P1376 and P36 symmetry evaluation | symmetry: 23.71%; 25.17%; 33.40%; 33.64% | Hurt |
| Edit only the MLP association without editing attention | ROME · ParaRel factual associations | generalization across expressions | Hurt |
Surprising findings
- Suppressing the plural neuron also reduces probability of non-number-specifying modifiers such as 'scattered', suggesting a co-occurrence frequency effect rather than grammatical rules.
- Causal tracing's division of labour between MLP and attention sites is not discernible for syntactic phenomena and is unstable for many factual traces.
Limitations
- Decoder-only models were evaluated on fewer BLiMP paradigms because target-before-source paradigms are unsuitable for them.
- ZsRE and CounterFact results are cited from prior work rather than re-run.
- Circuit-based alternative is only proposed and not formalised or tested.
Experimental setup
Datasets: BLiMP ParaRel
Benchmarks: BLiMP ParaRel
Metrics: attribution score R_1^2 localisation similarity reliability symmetry synonymy probability change
Baselines: KN edit ROME
Builds on: Knowledge Neurons ROME causal tracing BLiMP
Closest papers by meaning
- Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons (AAAI 2024 · similarity 0.91)
- Knowledge Localization: Mission Not Accomplished? Enter Query Localization! (ICLR 2025 · similarity 0.89)
- Towards Neuron Attributions in Multi-Modal Large Language Models (NeurIPS 2024 · similarity 0.88)
- Precise Localization of Memories: A Fine-grained Neuron-level Knowledge Editing Technique for LLMs (ICLR 2025 · similarity 0.88)
- Knowledge Circuits in Pretrained Transformers (NeurIPS 2024 · similarity 0.88)
- How Do Large Language Models Acquire Factual Knowledge During Pretraining? (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.