Research map / Large language models / Instruction and fine-tuning
Eliciting Harmful Capabilities by Fine-Tuning on Safeguarded Outputs
Jackson Kaunismaa, John Hughes, Christina Q Knight, Avery Griffin, Mrinank Sharma, Erik Jones
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune on Claude 3.5 Sonnet benign chemistry outputs instead of weak-only or textbook data | weak open-source models · 8 chemical weapons tasks | APGR: 38.8% uplift for Llama 3.3 70B, 24.7% for Llama 3.1 8B | Improved |
| use newer frontier model (Claude 4 Opus) to generate data | Llama 3.3 70B elicitation attack · anchored comparison | APGR: 71.1% uplift relative to Claude 3.5 Sonnet upper bound | Improved |
| increase fine-tuning dataset size up to 10,000 datapoints (ablation) | Llama 3.3 70B elicitation attack · anchored comparison per task | PGR: some tasks improve up to 10,000 datapoints, others saturate with 500 | Mixed |
| fine-tune on organic chemistry synthesis instead of other domains (ablation) | Llama 3.3 70B elicitation attack · anchored comparison | APGR: 33.7 vs 7.4 (inorganic chem synthesis), 11.2 (inorganic chemistry), 16.9 (biology), 17.7 (science/engineering) | Improved |
| use harmful chemical weapons dataset instead of safeguard-constrained benign dataset (ablation) | Llama 3.3 70B elicitation attack · anchored comparison | APGR: 33.7% -> 50.9% | Improved |
| generate data with classifier-guarded system instead of unguarded system (ablation) | Llama 3.3 70B elicitation attack · anchored comparison | APGR: 47.2% -> 49.0% | No clear effect |
| fine-tune on benign chemical data (route recovery) | Llama 3.3 70B · Task 5 synthesis route recovery | correct route rate: 0% -> 11% | Improved |
| fine-tune on benign chemical data (route recovery) | Llama 3.3 70B · Task 4 synthesis route recovery | correct route rate: 23% -> 0% | Hurt |
| use anchored comparison instead of rubrics for evaluation (ablation) | uplift evaluation · deliberately introduced mistakes | mistakes identified: 10.5% -> 50.9% | Improved |
Closest papers by meaning
- Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation (ICML 2024 · similarity 0.86)
- Scaling Trends for Data Poisoning in LLMs (AAAI 2025 · similarity 0.86)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.85)
- On Evaluating the Durability of Safeguards for Open-Weight LLMs (ICLR 2025 · similarity 0.85)
- Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs (NeurIPS 2026 · similarity 0.84)
- The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
