Research map / Large language models / Language, safety and interpretability

Stealth edits to large language models

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a theory of model editing based on the intrinsic dimension of feature vectors, along with in-place stealth edits, a jet-pack edit block, and stealth attacks on LLMs that inject a trigger-activated response without retraining.

Problem

Editing LLMs to correct specific hallucinations without altering other behaviour lacks a theoretical account of what determines editability and selectivity; this also exposes a vulnerability to stealthy backdoor-like attacks.

Approach

Shows that the separability-based intrinsic dimension of normalised feature vectors bounds the false-activation probability of a linear detector neuron, so one metric predicts both edit selectivity and attack susceptibility; uses this to design a jet-pack block with recentred RMSNorm and to implant edits by replacing one neuron in W1 and optimising a column of W2.

Key results

• In-place stealth edits are highly selective in intermediate and later layers: virtually no other dataset prompts activated the detector neuron in those layers.
• Jet-pack blocks built for 1,000 or 4,000 MCF hallucinations show extremely low false positive and perplexity ratios, with the paper stating scaling to 10,000 edits.
• Stealth attacks with corrupted or contextual triggers can implant attacker-chosen responses with little impact on other prompts, with theory-based bounds on false activation.

Measured effects reported in the paper

Surprising findings

• Intrinsic dimension of feature vectors is much lower than the ambient dimension, and higher editability implies higher vulnerability to stealth attacks.
• Edits perform best when inserted halfway through the model, contrary to Transformer-Patcher which places edits in the last layer.

Limitations

• Evaluation limited to MCF and ZsRE with three model families due to computational cost; only 1,000/500/300 samples used per experiment.
• Edits remove a neuron with smallest l1 norm in W1 which causes some perplexity change in later layers from pruning rather than the edit itself.
• Assumes negligible variation of bias-direction projection for Llama and Mamba, which is only approximate.

Experimental setup

Datasets: Multi-CounterFact ZsRE Wikipedia

Benchmarks: Multi-CounterFact ZsRE

Metrics: edit success rate perplexity ratio detector false positive rate theoretical guaranteed FPR attack false positive rate

Baselines: GRACE Transformer-Patcher ROME MEMIT

Builds on: ROME MEMIT GRACE Transformer-Patcher RMSNorm

Closest papers by meaning

• BadEdit: Backdooring Large Language Models by Model Editing (ICLR 2024 · similarity 0.89)
• Concept-ROT: Poisoning Concepts in Large Language Models with Model Editing (ICLR 2025 · similarity 0.89)
• Position: Editing Large Language Models Poses Serious Safety Risks (ICML 2025 · similarity 0.88)
• Should We Really Edit Language Models? On the Evaluation of Edited Language Models (NeurIPS 2024 · similarity 0.88)
• BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models (NeurIPS 2025 · similarity 0.88)
• Certifying Language Model Robustness with Fuzzed Randomized Smoothing: An Efficient Defense Against Backdoor Attacks (ICLR 2025 · similarity 0.87)