Research map / Large language models / Alignment and preferences
Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProFS, a tuning-free weight-editing method that projects MLP-value weights away from a toxic subspace found by SVD of centered toxic-minus-non-toxic embeddings, showing it is more sample-efficient and label-noise robust than DPO for toxicity reduction.
Problem
Preference-tuning alignment methods such as DPO are computationally expensive, need large preference datasets, lack transparency, and are sensitive to label noise, which is problematic for toxicity reduction in LLMs.
Approach
Identify a low-rank toxic subspace from SVD of centered embedding differences between toxic and non-toxic texts (centering removes the corpus-mean/stopword direction), then orthogonally project MLP-value weights away from that subspace in one closed-form edit; theoretically framed via factor analysis and shown to be a denoised single DPO gradient step.
Key results
- On GPT-2 medium, ProFS (N=500) reduces toxicity from 48.00% to 26.83% vs DPO (N=2000) 36.36%, with perplexity 32.50 vs 29.86 for DPO.
- With 50 label-flipped... DPO toxicity rises from 37.61% to 51.40% at 50% flipped labels while ProFS stays around 26.5-28.2%.
- ProFS reduces toxicity with as few as 5 datapoints and achieves significant reductions with 50 datapoints, whereas DPO needs far more data; on HH-Golden with Mistral-7B, ProFS win rate 78.2% vs DPO 74.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ProFS instead of DPO for toxicity alignment | GPT-2 Medium · RealToxicityPrompts challenge | toxicity: 36.36 -> 26.83 | Helped |
| Use ProFS instead of DPO for toxicity alignment | Mistral 7B · RealToxicityPrompts challenge | toxicity: 36.42 -> 30.40 | Helped |
| Use ProFS instead of DPO for toxicity alignment | Mistral-SFT 7B · RealToxicityPrompts challenge | toxicity: 23.96 -> 26.03 | Hurt |
| Use ProFS instead of DPO for toxicity alignment | OPT 6.7B · RealToxicityPrompts challenge | toxicity: 45.31 -> 43.49 | Helped |
| Use ProFS instead of DPO for toxicity alignment | GPT-J 6B · RealToxicityPrompts challenge | toxicity: 43.67 -> 37.36 | Helped |
| Omit centering of the preference matrix (ablation) | ProFS on GPT-2 Medium · WikiText-2 dev | perplexity: 32.50±0.28 -> 94.89±14.51 | Hurt |
| Omit centering of the preference matrix (ablation) | ProFS on GPT-2 Medium · RealToxicityPrompts challenge | toxicity: 26.83±0.89 -> 9.91±3.50 | Helped |
| Flip preference labels in 50% of samples (ablation) | ProFS on GPT-2 Medium · RealToxicityPrompts challenge | toxicity: 26.83±0.89 -> 28.15±1.48 | No clear effect |
| Flip preference labels in 50% of samples (ablation) | DPO on GPT-2 Medium · RealToxicityPrompts challenge | toxicity: 37.61±1.03 -> 51.40±0.56 | Hurt |
| Align on 500 HH-Golden samples with ProFS instead of DPO | Mistral 7B · HH-Golden | win rate: 74.1 -> 78.2 | Helped |
| Edit higher layers only (ablation) | ProFS · GPT-2 Medium | toxicity and perplexity | Mixed |
Surprising findings
- Flipping labels of up to 50% of preference pairs leaves ProFS toxicity essentially unchanged, since the Gram matrix is invariant to sign flips of rows.
- Omitting centering (keeping the corpus mean direction) causes near-nonsensical generations and high perplexity (94.89), showing the mean direction is critical to capability.
Limitations
- Editing relies on unsupervised subspace selection from activations, which is highly sensitive to the choice of singular vectors and rank k.
- Analysis and method restricted to MLP layers; self-attention not edited.
- Layer start L0 and rank k chosen via cross-validation and heuristics (ScreeNot), with different settings per model.
Experimental setup
Datasets: DPO-Toxic WikiText-2 RealToxicityPrompts HH-Golden PKU-Safe RLHF BoolQ RTE HellaSwag WinoGrande ARC OpenbookQA
Benchmarks: RealToxicityPrompts (challenge) WikiText-2 perplexity seven zero-shot tasks from LM Evaluation Harness HH-Golden win rate
Metrics: toxicity (Detoxify score) perplexity zero-shot capability average win rate
Baselines: DPO KTO DExperts Toxification Reversal
Builds on: DPO linear representation hypothesis ScreeNot
Closest papers by meaning
- A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity (ICML 2024 · similarity 0.92)
- Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization (ICLR 2025 · similarity 0.90)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.90)
- Provably Robust DPO: Aligning Language Models with Noisy Feedback (ICML 2024 · similarity 0.90)
- ROPO: Robust Preference Optimization for Large Language Models (ICML 2025 · similarity 0.89)
- Less is More: Improving LLM Alignment via Preference Data Selection (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.