Research map / Large language models / Reasoning and chain of thought
Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ArgRE, a test-time detoxification method that interpolates toxic/non-toxic hidden states along a PCA non-toxic direction to create dense transition data, trains a token-level autoregressive reward MLP, and edits representations with directional steering plus a few gradient steps.
Problem
LLMs generate toxic content; existing test-time representation editing methods rely on sparse toxicity annotations and make imprecise interventions because they poorly explore transitions between toxic and non-toxic outputs.
Approach
Model toxicity transitions by linearly interpolating final-layer hidden states between non-toxic and toxic responses along a PCA-derived non-toxic direction, turning sparse pairs into dense supervision for a token-level autoregressive reward model, which then drives an adaptive two-step representation edit (directional steering by expected reward gap, then a few gradient-ascent steps).
Key results
- ArgRE reduces toxicity by up to 62.21% across eight LLMs, versus 42.98% for GenARM, 27.88% for ProFS and 25.53% for Re-Control.
- ArgRE (w/ iter) inference time on LLaMA-30B is 9.30s vs 18.94s for GenARM, a 47.58% reduction.
- On jailbreak mitigation with Mistral 7B, ArgRE reaches 73.0% RtA versus 45.1% for the original model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ArgRE instead of GenARM | LLaMA-7B · RealToxicityPrompts challenge | Toxic: 23.86 (0.84) -> 18.06 (0.68) | Helped |
| Add iterative gradient refinement to the directional steering step (ablation) | ArgRE on LLaMA-7B · RealToxicityPrompts challenge | Toxic: 19.49 (0.59) -> 18.06 (0.68) | Helped |
| Add iterative gradient refinement to the directional steering step (ablation) | ArgRE on LLaMA-30B · 128 generated tokens per prompt | inference time (s): 8.20 -> 9.30 | Hurt |
| Use ArgRE instead of GenARM | LLaMA-30B · 128 generated tokens per prompt | inference time (s): 18.94 -> 9.30 | Helped |
| Use ArgRE instead of GenARM | Mistral 7B · Stereotype Recognition | accuracy: 53.5 -> 54.7 | Helped |
| Use ArgRE instead of GenARM | Mistral 7B · JailbreakTrigger | RtA rate: 68.4 -> 73.0 | Helped |
| Increase the number of toxicity transition trajectories from 0 to 7 (ablation) | ArgRE on LLaMA-7B · RealToxicityPrompts challenge | Toxic: reduces Toxic by 8.58% | Helped |
| Increase the number of toxicity transition trajectories from 7 to 15 (ablation) | ArgRE on LLaMA-7B · RealToxicityPrompts challenge | Toxic: marginal gain of 0.24% | Helped |
| Use one interpolated toxicity transition instead of no transitions (ablation) | ArgRE on LLaMA-7B · RealToxicityPrompts challenge | Toxic: 2.8% lower toxicity score than GenARM at N_in = 0 | Helped |
| Increase gradient-refinement step size from η=0 to η=1.0 (ablation) | ArgRE on LLaMA-7B · RealToxicityPrompts challenge | Toxic: 19.49 -> 17.58 | Helped |
| Increase gradient-refinement step size from η=0 to η=1.0 (ablation) | ArgRE on LLaMA-7B · RealToxicityPrompts challenge | PPL_g: 11.60 -> 12.66 | Hurt |
| Use ArgRE with 100 toxicity annotations instead of 2000 (ablation) | ArgRE on LLaMA-7B · RealToxicityPrompts challenge | Toxic: 43.27% to 22.19% | Helped |
Surprising findings
- Raising N_in from 0 to 1 interpolated transition is enough for ArgRE to surpass GenARM, which was the strongest baseline, indicating transitions add substantial supervision value.
Limitations
- White-box method requiring access to internal representations of the LLM.
- Toxicity transition exploration uses only the first principal component direction.
Experimental setup
Datasets: pairwise toxic dataset (Wikitext-2 non-toxic, PPLM toxic) WikiText-2 RealToxicityPrompts TrustLLM stereotype and jailbreak data
Benchmarks: RealToxicityPrompts WikiText-2 BoolQ RTE HellaSwag WinoGrande ARC OpenbookQA TrustLLM stereotype recognition JailbreakTrigger
Metrics: Detoxify toxicity score perplexity zero-shot accuracy refuse-to-answer rate inference time
Baselines: ProFS Re-Control GenARM DPO Self-Detoxify DeStein
Builds on: ProFS Re-Control GenARM DPO linear representation hypothesis
Closest papers by meaning
- UniDetox: Universal Detoxification of Large Language Models via Dataset Distillation (ICLR 2025 · similarity 0.91)
- Large Language Models can Become Strong Self-Detoxifiers (ICLR 2025 · similarity 0.91)
- Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity (ICLR 2025 · similarity 0.88)
- IF-Guide: Influence Function-Guided Detoxification of LLMs (NeurIPS 2025 · similarity 0.87)
- POROver: Improving Safety and Reducing Overrefusal in Large Language Models with Overgeneration and Preference Optimization (ICML 2025 · similarity 0.86)
- When Bad Data Leads to Good Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.