Research map / Large language models / Retrieval and knowledge

Golden Layers and Where to Find Them: Improved Knowledge Editing for Large Language Models Via Layer Gradient Analysis

NeurIPS 2026 · Large language models › Retrieval and knowledge · approach: Knowledge editing in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes the "golden layer" hypothesis for knowledge editing (a fixed layer that near-matches per-sample optimal layers) and Layer Gradient Analysis (LGA), a first-order gradient-attribution method that selects the golden layer from a small proxy set, outperforming CMA and SaLEM with R-ROME, ROME and EMMET.

Problem

Choosing which layer to edit in locate-then-edit knowledge editing is critical, but causal tracing (CMA) often fails to identify the best layer and per-sample or per-layer trial-and-error is computationally expensive.

Approach

Observes that a fixed "golden" layer achieves near-optimal aggregate editing performance versus per-sample optimal layers, and estimates it by summing layer-restricted gradient inner products between old-knowledge and new-knowledge losses over a proxy set of queries, with no actual edits needed.

Key results

• On GPT-2 XL with R-ROME, LGA improves Overall over CMA on ZSRE (0.7106 vs 0.6942) and WikiCounterfact (0.6808 vs 0.6741), and gains of 8.56% Rewrite Accuracy over CMA on WikiBio.
• Golden layers from the proxy set generalize to test sets; no statistically significant difference from sample-wise optimal layers in 5 of 15 model-dataset cases (Rewrite Accuracy).
• LGA reports speedups of roughly 30x to over 60x relative to brute-force golden layer search on GPT-2 XL.

Surprising findings

• For Gemma3-12B on WikiBio, the golden layer falls well short of sample-wise optimal Rewrite Accuracy (0.8538 vs 0.9296), and CMA trails LGA by a wide margin.
• EMMET with LGA underperforms CMA on WikiBio Overall (0.6707 vs 0.6921).

Limitations

• Gains over CMA are modest on some metrics and datasets, and LGA trails CMA on WikiBio with EMMET.
• Golden layers are model-dependent and the proxy sets are small (e.g., ~30 samples for WikiBio), so estimates may be noisy.
• Evaluated only on three LLMs and MLP-layer editing methods (ROME, R-ROME, EMMET).

Experimental setup

Datasets: ZSRE WikiBio WikiCounterfact WikiRecent Counterfact

Benchmarks: ZSRE WikiBio WikiCounterfact WikiRecent Counterfact

Metrics: Rewrite Accuracy Rephrase Accuracy Locality Portability Fluency Overall

Baselines: CMA SaLEM random layer selection brute-force golden layer search

Builds on: R-ROME ROME MEMIT EMMET causal mediation analysis EasyEdit

Closest papers by meaning

• Uncovering Overfitting in Large Language Model Editing (ICLR 2025 · similarity 0.90)
• UniEdit: A Unified Knowledge Editing Benchmark for Large Language Models (NeurIPS 2025 · similarity 0.90)
• AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025 · similarity 0.90)
• LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions (NeurIPS 2025 · similarity 0.89)
• WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models (NeurIPS 2024 · similarity 0.89)
• In-Context Editing: Learning Knowledge from Self-Induced Distributions (ICLR 2025 · similarity 0.89)