Research map / Large language models / Retrieval and knowledge
Identifying Query-Relevant Neurons in Large Language Models for Long-Form Texts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QRNCA, an architecture-agnostic gradient-based attribution method that locates query-relevant neurons in decoder-only LLMs for long-form answers by reformulating queries as multiple-choice QA, with inverse cluster attribution and common-neuron removal.
Problem
Locating knowledge neurons for long-form, free-form answers in contemporary decoder-only 7B LLMs, where prior methods were limited to single-token triplet facts in small encoder or GPT-2-scale models.
Approach
Transform free-form queries into multiple-choice QA so the model outputs only an option letter, compute integrated-gradient neuron attribution over FFN up-projection activations, normalize across clusters with inverse cluster attribution (down-weighting neurons shared across queries), and remove common neurons tied to option letters and frequent tokens.
Key results
- On Llama-2-7B language dataset, QRNCA reaches a boost PCR of 41.2 and suppress PCR of 36.0, versus 6.7 and 1.8 for Knowledge Neuron*.
- Knowledge editing success rate on language data is 16.6% (boost) and 24.8% (suppress) for QRNCA, versus 14.3% and 16.0% for Knowledge Neuron*.
- Neuron-based prediction on MMLU-derived biology reaches 0.96 accuracy, matching the prompt-based 0.96, though chemistry is lower (0.67 vs 0.71).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Inverse Cluster Attribution to QRNCA (ablation) | QRNCA · Domain dataset, boost | PCR: 2.5 -> 4.4 | Helped |
| Add Inverse Cluster Attribution to QRNCA (ablation) | QRNCA · Domain dataset, suppress | PCR: 1.1 -> 5.6 | Helped |
| Add Inverse Cluster Attribution to QRNCA (ablation) | QRNCA · Language dataset, boost | PCR: 6.5 -> 41.2 | Helped |
| Add Inverse Cluster Attribution to QRNCA (ablation) | QRNCA · Language dataset, suppress | PCR: 2.2 -> 36.0 | Helped |
| Remove common neurons (ablation) | QRNCA · Domain dataset, boost | PCR: 2.8 -> 4.4 | Helped |
| Remove common neurons (ablation) | QRNCA · Domain dataset, suppress | PCR: 1.8 -> 5.6 | Helped |
| Remove common neurons (ablation) | QRNCA · Language dataset, boost | PCR: 10.4 -> 41.2 | Helped |
| Remove common neurons (ablation) | QRNCA · Language dataset, suppress | PCR: 8.5 -> 36.0 | Helped |
| Use neuron-based prediction instead of prompt-based prediction | Llama-2-7B · MMLU validation, Chemistry | Accuracy: 0.71 -> 0.67 | Hurt |
Surprising findings
- Domains show higher neuron overlap rates than languages, suggesting polysemantic storage of domain concepts in single neurons.
- Common neurons predominantly express option letters and stop words and sit in the top layer, and the highest-scoring neurons in middle layers do not yield human-readable tokens under vocabulary projection.
Limitations
- Evaluated only on 7B-scale models (Llama-2-7B and Mistral-7B) and relies on hand-constructed multiple-choice proxy tasks.
- Hyperparameters (threshold, top-v, common-neuron frequency) were tuned on a small biology hold-out set of 50 samples.
- Middle-layer neurons do not yield interpretable token projections, so the semantic meaning of detected neurons remains unclear.
Experimental setup
Datasets: MMLU (six high-school subjects) Multilingual LAMA (six languages)
Benchmarks: Domain multi-choice QA dataset (derived from MMLU) Language multi-choice QA dataset (derived from Multilingual LAMA)
Metrics: Probability Change Ratio (PCR) Probability change percentage (boost/suppress) knowledge editing success rate neuron-based prediction accuracy
Baselines: Random Neuron Activation Knowledge Neuron* QRNCA w/o ICA QRNCA w/ Common Neuron
Builds on: Knowledge Neurons (Dai et al., 2022) ROME Logit Lens MMLU
Closest papers by meaning
- Knowledge Localization: Mission Not Accomplished? Enter Query Localization! (ICLR 2025 · similarity 0.90)
- Precise Localization of Memories: A Fine-grained Neuron-level Knowledge Editing Technique for LLMs (ICLR 2025 · similarity 0.90)
- Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neurons (AAAI 2024 · similarity 0.89)
- How do Large Language Models Handle Multilingualism? (NeurIPS 2024 · similarity 0.89)
- Towards Neuron Attributions in Multi-Modal Large Language Models (NeurIPS 2024 · similarity 0.88)
- WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.