Research map / Large language models / Instruction and fine-tuning
Tamper-Resistant Safeguards for Open-Weight LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TAR, a meta-learning-style adversarial training method that makes safeguards (knowledge unlearning on WMDP and harmful-request refusal) on open-weight Llama-3-8B-Instruct resist hundreds of fine-tuning attacks while largely preserving benign capabilities.
Problem
Open-weight LLM safeguards (refusal and unlearning) are removed by a few steps of fine-tuning, so they are not tamper-resistant against adversaries who can modify model weights.
Approach
Two-phase method: first install an initial safeguard (Random Mapping for knowledge restriction, refusal training for refusal), then run adversarial training in which simulated multi-step SFT attacks are unrolled with a first-order (straight-through) meta-learning approximation and the outer loop maximizes an entropy (or DPO-based) tamper-resistance loss on the attacked model, with an representatio
Key results
- On Llama-3-8B, TAR averaged post-attack WMDP Biosecurity forget accuracy of 35.2% vs 64.9% for RMU and 61.3% for LLMU, with retain MMLU dropping to 54.7%.
- Across 26 biosecurity fine-tuning adversaries, TAR keeps post-attack forget accuracy near random chance for nearly all attacks where baselines break down.
- For harmful request refusal, TAR reduces average post-attack HarmBench ASR to 63.9% vs 72.5% for refusal-trained Llama-3-8B-Instruct and 84.8% for RR, at a cost of 1.7 MT-Bench points (8.0 to 6.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add TAR tamper-resistance training to the initial knowledge-restriction safeguard | Llama-3-8B-Instruct · WMDP Biosecurity, Chemical Security, and Cybersecurity | post-attack forget accuracy: 70.5 -> 35.2; 47.8 -> 27.1; 46.4 -> 28.6 | Helped |
| Add TAR tamper-resistance training to refusal training | Llama-3-8B-Instruct · HarmBench | post-attack ASR: 72.5 -> 63.9 | Helped |
| Add TAR refusal safeguard | Llama-3-8B-Instruct · MT-Bench | pre-attack score: 8.1 -> 6.3 | Hurt |
| Add adversarial training after the initial Random Mapping safeguard (ablation) | TAR weaponization safeguard · WMDP Biosecurity | post-attack forget accuracy: 61.6 -> 35.2 | Helped |
| Add the initial Random Mapping safeguard before adversarial training (ablation) | TAR weaponization safeguard · WMDP Biosecurity | pre-attack forget accuracy: 47.3 -> 28.1 | Helped |
| Increase tamper-resistance loss weight from λTR=1.0 to λTR=4.0 (ablation) | TAR weaponization safeguard · WMDP Biosecurity | post-attack forget accuracy: 39.9 -> 35.2 | Helped |
| Increase tamper-resistance loss weight from λTR=1.0 to λTR=4.0 (ablation) | TAR weaponization safeguard · MMLU retain set | pre-attack retain accuracy: 62.5 -> 54.7 | Hurt |
| Include the retain-set SFT followed by Biosecurity SFT adversary during TAR training (ablation) | TAR weaponization safeguard · WMDP Biosecurity | post-attack forget accuracy: 44.9 -> 35.2 | Helped |
| Include the retain-set SFT followed by Biosecurity SFT adversary during TAR training (ablation) | TAR weaponization safeguard · R→F attack | post-attack accuracy: 58.7 -> 29.7 | Helped |
| Increase the simulated adversary inner-loop length K (ablation) | TAR weaponization safeguard · 1,000-step biosecurity SFT attack | test-loss plateau magnitude and duration | Helped |
| Use targeted train-time SFT adversaries to patch observed vulnerabilities (ablation) | TAR weaponization safeguard · WMDP Biosecurity | robustness to test-time attacks: 44.9 -> 35.2; 58.7 -> 29.7 | Helped |
Surprising findings
- Maximizing cross-entropy as the tamper-resistance loss causes the adversary to recover quickly; negative entropy loss is needed to flatten the adversary's loss trajectory.
- Post-attack retain accuracy is also low for TAR, reflecting a trade-off that the authors argue is acceptable since the attacker must preserve it.
Limitations
- Reduces retain (MMLU) accuracy substantially (about 10.6 points on average), and post-attack retain accuracy is also low, complicating benign fine-tuning.
- Varying robustness to PEFT and some out-of-distribution learning-rate attacks; focuses only on SFT attacks rather than other tampering methods.
- Scaling to larger models poses computational challenges.
Experimental setup
Datasets: WMDP forget sets (biosecurity, chemical, cybersecurity) MMLU retain subjects Anthropic-HH-RLHF Magpie-Align WMDP auxiliary economics corpora
Benchmarks: WMDP MMLU HarmBench MT-Bench
Metrics: accuracy (WMDP forget and retain) post-attack forget accuracy attack success rate MT-Bench score
Baselines: RMU LLMU Max Entropy Min Posterior Representation Rerouting R2D2 RepNoise Refusal Trained Llama-3-8B-Instruct
Builds on: RMU MAML Representation Rerouting DPO Circuit breaking
Closest papers by meaning
- On Evaluating the Durability of Safeguards for Open-Weight LLMs (ICLR 2025 · similarity 0.92)
- CoreGuard: Safeguarding Foundational Capabilities of LLMs Against Model Stealing in Edge Deployment (NeurIPS 2025 · similarity 0.89)
- MoGU: A Framework for Enhancing Safety of LLMs While Preserving Their Usability (NeurIPS 2024 · similarity 0.87)
- Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space (NeurIPS 2024 · similarity 0.87)
- Representation Noising: A Defence Mechanism Against Harmful Finetuning (NeurIPS 2024 · similarity 0.87)
- Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.