Research map / Large language models / Alignment and preferences
A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Mechanistically studies how DPO reduces toxicity in GPT2-medium, finding that DPO bypasses rather than removes toxic MLP value vectors via a distributed residual-stream offset, and that scaling toxic key vectors reverts the model to toxic behavior.
Problem
Alignment algorithms like DPO suppress toxic behavior, but the mechanisms by which models become aligned are not understood, which makes jailbreaks and un-alignment hard to explain.
Approach
Locate toxicity-promoting MLP value vectors in GPT2 via a linear probe, apply DPO on PPLM-generated toxic/non-toxic pairs, and show that DPO leaves all weights nearly unchanged but learns a distributed residual-stream offset that moves activations out of toxic key-vector regions; re-amplifying those key vectors un-aligns the model.
Key results
- DPO reduces mean toxicity on RealToxicityPrompts challenge from 0.453 (GPT2) to 0.208 with perplexity 23.34 and F1 0.195.
- Subtracting toxic probe vector reduces toxicity to 0.245 with perplexity 23.56; every GPT2 parameter in DPO model has cosine similarity above 0.99 with the original.
- Scaling the key vectors of 7 toxic MLP neurons by 10x in the DPO model restores toxicity to 0.458, matching pre-aligned GPT2 (0.453).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DPO using toxic/nontoxic preference pairs | GPT2-medium · RealToxicityPrompts challenge | toxicity: 0.453 -> 0.208 | Helped |
| Subtract the toxicity probe vector from the last-layer residual stream | GPT2-medium · RealToxicityPrompts challenge | toxicity: 0.453 -> 0.245 | Helped |
| Subtract MLP.v770 at layer 19 from the last-layer residual stream | GPT2-medium · RealToxicityPrompts challenge | toxicity: 0.453 -> 0.305 | Helped |
| Subtract toxic SVD vector 0 from the last-layer residual stream | GPT2-medium · RealToxicityPrompts challenge | toxicity: 0.453 -> 0.268 | Helped |
| Apply DPO using toxic/nontoxic preference pairs | GPT2-medium · Wikipedia sentences | F1: 0.193 -> 0.195 | No clear effect |
| Scale 7 toxic key vectors by 10x | GPT2 DPO · RealToxicityPrompts challenge | toxicity: 0.208 -> 0.458 | Hurt |
| Scale 7 toxic key vectors by 10x | GPT2 DPO · Wikitext-2 | perplexity: 23.34 -> 23.30 | No clear effect |
Surprising findings
- Toxic MLP value vectors are essentially unchanged by DPO; the model avoids toxicity by shifting the residual stream so toxic key vectors are not activated.
- The distributed offset comes from earlier-layer value-vector shifts that point in the opposite direction of the residual shift, contributing because GeLU inactive neurons have small negative activations.
Limitations
- Single small model (GPT2-medium) and a single toxicity case study
- Toxicity measured with Perspective API on the challenge subset only; un-alignment demonstrated with one simple method
Experimental setup
Datasets: Jigsaw toxic comment classification Wikitext-2 RealToxicityPrompts
Benchmarks: RealToxicityPrompts challenge subset Wikitext-2 perplexity Wikipedia continuation F1
Metrics: toxicity (Perspective API) perplexity F1
Baselines: GPT2 (no op) subtracting toxic probe vector subtracting MLP.v770^19 subtracting SVD toxic vector 0
Builds on: DPO PPLM logit lens Geva et al. value vector interpretation
Closest papers by meaning
- Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity (ICLR 2025 · similarity 0.92)
- Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO (NeurIPS 2025 · similarity 0.90)
- Improving LLM Safety Alignment with Dual-Objective Optimization (ICML 2025 · similarity 0.89)
- Is Poisoning a Real Threat to DPO? Maybe More So Than You Think (AAAI 2025 · similarity 0.89)
- Towards Efficient Exact Optimization of Language Model Alignment (ICML 2024 · similarity 0.89)
- Mission Impossible: A Statistical Perspective on Jailbreaking LLMs (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.