Research map / Large language models / Instruction and fine-tuning
Representation Noising: A Defence Mechanism Against Harmful Finetuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Representation Noising (RepNoise), a defence that removes information about harmful representations across all layers of an LLM so that harmful fine-tuning attacks become much harder, even when attackers hold the weights.
Problem
Open-weight or stolen LLMs can be harmfully fine-tuned by attackers who control the weights, and existing safety guardrails and safety fine-tuning are easily reversed by such harmful fine-tuning attacks (HFAs).
Approach
Pre-attack, the defender trains the model with a loss combining stability on harmless data, layer-wise gradient ascent on harmful language modelling loss, and an MMD-based term pushing harmful intermediate activations toward Gaussian noise, removing information about harmful representations across all layers rather than only suppressing outputs.
Key results
- RepNoise reduces harmfulness score after 1k-sample attacks at learning rates 3e-5 to 8e-5 to 0.08-0.11, versus 0.47-0.74 for the base llama2-7b-chat, and is the only defence consistently resistant across attacks.
- On toxic content generation (RTP), RepNoise keeps Perspective toxicity at 0.00-0.07 after attacks, versus 0.40-0.74 for the base model.
- RepNoise preserves general capability: TruthfulQA 0.37 vs 0.38, MMLU 0.45 vs 0.46 for base, and retains trainability on GEM tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply RepNoise before harmful fine-tuning | Llama 2 7B Chat · BeaverTails harmful QA across the reported learning rates and sample counts | harmfulness score: 0.47 -> 0.08; 0.74 -> 0.12; 0.73 -> 0.10; 0.72 -> 0.13; 0.74 -> 0.11; 0.73 -> 0.12 | Helped |
| apply RepNoise before fine-tuning | Llama 2 7B Chat · RTP toxic generation, 3×10^-5 | toxicity score: 0.40 -> 0.00 | Helped |
| apply RepNoise before fine-tuning | Llama 2 7B Chat · RTP toxic generation, 6×10^-5 | toxicity score: 0.74 -> 0.05 | Helped |
| apply RepNoise before fine-tuning | Llama 2 7B Chat · RTP toxic generation, 8×10^-5 | toxicity score: 0.71 -> 0.07 | Helped |
| apply RepNoise | Llama 2 7B Chat · TruthfulQA | score: 0.38 -> 0.37 | No clear effect |
| apply RepNoise | Llama 2 7B Chat · MMLU | score: 0.46 -> 0.45 | No clear effect |
| apply RepNoise | Llama 2 7B Chat · Hellaswag | score: 0.58 -> 0.57 | No clear effect |
| apply RepNoise | Llama 2 7B Chat · Winogrande | score: 0.66 -> 0.66 | No clear effect |
| apply RepNoise | Llama 2 7B Chat · BeaverTails harmful QA with withheld harm categories and unseen samples | harmfulness score: Base: 0.49, 0.51, 0.40, 0.52, 0.53, 0.35; RepNoise: 0.08, 0.05, 0.06, 0.09, 0.01, 0.08 | Helped |
| apply additional safety training | Llama 2 7B Chat · BeaverTails harmful QA attacks | harmfulness score | Hurt |
| replace RepNoise with adversarial loss | Llama 2 7B Chat · BeaverTails harmful QA, stronger attacks | harmfulness score: 0.70, 0.75, 0.77, 0.77 | Hurt |
| freeze the LM head during RepNoise (ablation) | RepNoise on Llama 2 7B Chat · BeaverTails harmful QA, 3×10^-5, 10k samples | harmfulness score: 0.12 -> 0.10 | No clear effect |
Surprising findings
- Fine-tuning on benign GEM tasks made the HFA more effective even against RepNoise, with a small increase after CACAPO training.
- RepNoise does not generalize out-of-distribution: defence on BeaverTails fails against HEX-PHI and toxic content attacks.
Limitations
- Defence can be defeated at higher learning rates and with more data, and is sensitive to hyperparameters.
- Requires paired safe and unsafe examples, making data collection costly.
- Does not generalize out-of-distribution to HEX-PHI or toxic content attacks.
Experimental setup
Datasets: BeaverTails DecodingTrust RealToxicityPrompts split GEM benchmark (ViGGO, E2E NLG, DART, CACAPO, ConvWeather) HEX-PHI
Benchmarks: BeaverTails harmfulness classifier Perspective API toxicity TruthfulQA MMLU Hellaswag Winogrande ARC-easy Ethics CrowS-Pairs GEM ROUGE-1
Metrics: harmfulness classifier score Perspective API toxicity score accuracy ROUGE-1
Baselines: Security Vectors Vaccine Additional safety training Gradient ascent Adversarial loss Random initialization
Builds on: Vaccine Achille et al. transfer-learning transition probability information bottleneck Rosati et al. immunization conditions tuned lens
Closest papers by meaning
- Panacea: Mitigating Harmful Fine-tuning for Large Language Models via Post-fine-tuning Perturbation (NeurIPS 2025 · similarity 0.90)
- Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack (ICML 2025 · similarity 0.90)
- Learning and Forgetting Unsafe Examples in Large Language Models (ICML 2024 · similarity 0.90)
- Safe LoRA: The Silver Lining of Reducing Safety Risks when Finetuning Large Language Models (NeurIPS 2024 · similarity 0.89)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.89)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.