Research map / Architectures and efficiency / RNNs and normalization

Norm Anchors Make Model Edits Last

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Layer-wise analysis of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Identifies a norm-feedback loop in sequential locate-and-edit model editing and proposes Norm-Anchor Scaling (NAS), a one-line rescaling of solved value vectors to an original-model reference norm, which extends the usable editing horizon over 4x.

Problem

Sequential locate-and-edit (L&E) knowledge editors abruptly fail after a few thousand edits, with edited-weight norms growing and capabilities collapsing.

Approach

Analyze sequential L&E as a positive norm-feedback loop (larger weights yield larger solved value vectors, which further enlarge weights) and break it by rescaling each solved value vector to a fixed reference norm measured on the unedited model, preserving its direction.

Key results

• Across four L&E editors on Llama3-8B, NAS extends collapse point by 3.75x for MEMIT and PRUNE (400 to 1.5k edits), 3.38x for RECT, and more than 5.25x for AlphaEdit (4k to over 21k).
• Average editing success rises from 51.9 to 89.3 (+37.4 pp, +72.2% relative) across editors and metrics at the collapse point.
• Single-edit efficacy is not reduced: AlphaEdit+NAS changes Eff. by +1.00 and MEMIT+NAS by 0.00 on 1,000 single CounterFact edits.

Surprising findings

• Existing increment-level norm controls (update regularization or clipping) do not resolve the feedback loop, since they constrain the update step rather than the solved value vector.

Limitations

• Uses a fixed original-model reference norm, which may be suboptimal when edit difficulty, locality requirements, or edit distribution change substantially.
• Scaling analysis relies on empirical linear scaling assumptions and an approximately constant key norm, not a fully general proof.
• Evaluated mainly on factual editing benchmarks with L&E editors; the GPT-J and Llama3-8B results dominate the analysis.

Experimental setup

Datasets: CounterFact ZsRE WikiBigEdit MQuAKE-CF-3k-v2 KnowEdit WikiRecent KnowEdit WikiBio GLUE (SST, MRPC, RTE, CoLA, MNLI) MMLU

Benchmarks: CounterFact ZsRE GLUE-style tasks MMLU WikiBigEdit MQuAKE-CF-3k-v2

Metrics: Efficacy Generalization Specificity Fluency Consistency Score (harmonic mean) collapse point CP@60 F1 on GLUE tasks norm ratio R_n

Baselines: MEMIT PRUNE RECT AlphaEdit FT UltraEdit RLEdit GRACE MELO WISE MEMOIR ENCORE

Builds on: ROME MEMIT AlphaEdit PRUNE RECT

Closest papers by meaning

• Rethinking Residual Distribution in Locate-then-Edit Model Editing (NeurIPS 2025 · similarity 0.89)
• Towards Lifelong Model Editing via Simulating Ideal Editor (ICML 2025 · similarity 0.87)
• BalancEdit: Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editing (ICML 2025 · similarity 0.86)
• Reasons and Solutions for the Decline in Model Performance after Editing (NeurIPS 2024 · similarity 0.85)
• Modality-Decoupled Online Recursive Editing (ICML 2026 · similarity 0.85)
• Spectral Editing of Activations for Large Language Model Alignment (NeurIPS 2024 · similarity 0.84)