Research map / Large language models / Language, safety and interpretability

Spectral Editing of Activations for Large Language Model Alignment

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SEA, a training-free inference-time activation editing method that uses SVD of cross-covariance between neutral and positive/negative demonstration activations (with optional invertible feature functions for non-linear editing) to improve truthfulness and reduce bias in LLMs.

Problem

LLMs generate untruthful or biased content; existing activation-editing methods typically need expensive iterative optimisation to learn the intervention target module.

Approach

Edit last-token MLP activations by projecting them onto principal directions that maximally covary with positive-demonstration activations and minimally covary with negative-demonstration activations, obtained in closed form via SVD of cross-covariance matrices; extend to non-linear editing through invertible feature functions (squared-exponential, tanh, ELU) with pseudo-inverses.

Key results

• On LLaMA-2-Chat-7B, linear SEA raises TruthfulQA MC1 from 36.96 to 39.41 with about 3.67% inference-time overhead.
• Non-linear Phi-SEA raises BBQ accuracy for LLaMA-2-7B from 43.0 to 56.2 and lowers stereotypical response rate from 52.0 to 42.6.
• Linear SEA improves TruthfulQA MC1 across all six tested LLMs (e.g., Mistral-7B 55.8 to 56.4), and only 25 demonstrations already yield gains.

Measured effects reported in the paper

Surprising findings

• Reversing the positive and negative editing projections drops MC1 to 35.13, below the unedited chat baseline, supporting that the projections capture behaviour-relevant directions.
• LoRA fine-tuning on the same data gives little or no improvement over LLaMA-2-Chat, unlike training-free SEA.

Limitations

• Non-linear SEA uses pseudo-inverses of feature functions that are not lossless, causing drops on control tasks such as HellaSwag and Natural Questions.
• Joint truthfulness and fairness editing is less effective than specialised editing for each target.
• BBQ linear SEA gains are modest (about 0.8 points), likely due to poor linear separability of positive and negative activations.

Experimental setup

Datasets: HaluEval BBQ

Benchmarks: TruthfulQA BBQ CrowS-Pairs MMLU HellaSwag Natural Questions GSM8K MathQA ToxiGen

Metrics: MC1 MC2 GPT-Truth GPT-Info Info*Truth accuracy unknown-answer rate stereotypical response rate bias score exact match

Baselines: In-context learning LoRA fine-tuning Inference-time Intervention (ITI) DoLa Contrastive Decoding (CD) Induce-then-Contrast Decoding (ICD)

Builds on: inference-time intervention (ITI) LoRA contrastive decoding

Closest papers by meaning

• Aligning Large Language Models with Representation Editing: A Control Perspective (NeurIPS 2024 · similarity 0.89)
• Resolving Lexical Bias in Model Editing (ICML 2025 · similarity 0.89)
• AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models (ICLR 2025 · similarity 0.88)
• Differentially Private Steering for Large Language Model Alignment (ICLR 2025 · similarity 0.88)
• Joint Localization and Activation Editing for Low-Resource Fine-Tuning (ICML 2025 · similarity 0.88)
• Rethinking Residual Distribution in Locate-then-Edit Model Editing (NeurIPS 2025 · similarity 0.87)