Research map / Large language models / Retrieval and knowledge
Capability Localization: Capabilities Can be Localized rather than Individual Knowledge
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that individual-knowledge neuron localization methods (KN, ROME, KC) are unfaithful and unreliable, then proposes Commonality Neuron Localization (CNL) that finds task-commonality neurons which act as cross-dataset capability neurons.
Problem
Where and how model parameters store knowledge and capabilities: existing individual-knowledge localization methods (KN, ROME causal tracing, KC) give inconsistent neuron sets for semantically identical prompts and their verification experiments are unreliable.
Approach
Instead of localizing individual facts, locate neurons whose attribution scores are shared across a dataset of same-type samples (commonality neurons, via expanded KN integrated gradients over generated tokens with a variance-based threshold), and show these neurons act as capability neurons that transfer across datasets when enhanced or erased.
Key results
- CNL achieves a neuron overlap rate of 96.42% on GSM8K for Llama2-7B, versus 37.3% for KN, 32.7% for ROME and 7.2% for KC on fidelity tests.
- Erasing CNL-located neurons (sigma=3) drops Llama2-13B GSM8K-related average performance by 18.68%, while random erasure changes performance by only about 0.2%.
- Fine-tuning only located neurons (0.15% of parameters) improves Llama2-7B average accuracy by 5.17% at epoch 1 and GPT-J-6B Emotion accuracy by 9.87% at epoch 10.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CNL instead of KN, ROME, or KC for neuron localization | GSM8K on Llama2-7B · GSM8K | Overlap: 37.3 -> 96.4; 32.7 -> 96.4; 7.2 -> 96.4 | Helped |
| Fine-tune the located neurons rather than randomly selected neurons | Llama2-7B · Avg. | accuracy: 22.47 -> 34.12 | Helped |
| Fine-tune the located neurons rather than mask them and fine-tune all others | Llama2-7B · Avg. | accuracy: 28.95 -> 34.12 | Helped |
| Fine-tune located neurons instead of randomly selected neurons | Llama2-7B · Avg. | accuracy: 24.43 -> 42.12 | Helped |
| Erase the located neurons (σ=3) (ablation) | Llama2-7B · Avg. | accuracy: 19.48 -> 8.62 ($\downarrow 10.86$) | Hurt |
| Erase the located neurons (σ=3) (ablation) | Llama2-13B · Avg. | accuracy: 26.10 -> 7.42 ($\downarrow 18.68$) | Hurt |
| Use GSM8K-located neurons for enhancement | Llama2-7B · Meta_Math | performance: 22\% | Helped |
| Enhance commonality neurons rather than randomly selected neurons | Llama2-7B, σ=6 · Other datasets not involved in training | average performance improvement: 3.20 -> 11.18 | Helped |
| Use commonality neurons rather than KN localization | CNL · GSM8K | IPP: 11.4 -> 20.0 | Helped |
Surprising findings
- Increasing or zeroing KN-localized activations often fails to change the target probability in the expected direction; random neurons produce similar effects.
- Editing randomly chosen non-localized layers achieves similar knowledge editing performance to the ROME-localized layer.
Limitations
- The main experiments use GPT-J-6B, Llama2-7B and Llama2-13B; the authors note that limited model scale (max 7B in the appendix) and model choice restrict generality.
- Commonality neuron overlap may partly reflect the shared answer format or multiple-choice/comprehension demands rather than a pure capability.
Experimental setup
Datasets: COUNTERFACT zsRE GSM8K Meta_Math Code25K Emotion Imdb
Benchmarks: GSM8K Emotion Code25K Meta_Math Imdb COUNTERFACT zsRE
Metrics: neuron overlap IoU neuron ratio accuracy IPP (impact on performance) probability change
Baselines: KN ROME KC Random neuron selection W/o located neuron fine-tuning
Builds on: Knowledge Neurons (KN) ROME causal tracing knowledge circuits (KC)
Closest papers by meaning
- Knowledge Localization: Mission Not Accomplished? Enter Query Localization! (ICLR 2025 · similarity 0.89)
- Precise Localization of Memories: A Fine-grained Neuron-level Knowledge Editing Technique for LLMs (ICLR 2025 · similarity 0.87)
- Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons (AAAI 2024 · similarity 0.87)
- Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency (NeurIPS 2026 · similarity 0.85)
- Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localization (ICML 2025 · similarity 0.85)
- Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.