Research map / Large language models / Instruction and fine-tuning
Estimating Worst-Case Frontier Risks of Open-Weight LLMs
Eric Wallace, Olivia Watkins, Miles Wang, Kai Chen, Chris Koch
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| anti-refusal incremental RL with a helpful-only reward | gpt-oss-120b · GPQA and unsafe prompts | capability and refusal rate: refusal rates near 0%, GPQA maintained | Improved |
| malicious fine-tuning RL with browsing and in-domain bio data | gpt-oss-120b · biology proxy evals (Gryphon, Tacit Knowledge, ProtocolQA, TroubleshootingBench) | accuracy | Improved |
| malicious fine-tuning RL on CTFs in a terminal environment | gpt-oss-120b · professional CTFs | pass@12 | No clear effect |
| add browsing tool during cyber RL (ablation) | gpt-oss-120b · CTF challenges | progress on tasks: browsing helped 4% of the time | No clear effect |
| consensus@k inference-time compute (ablation) | gpt-oss MFT bio · Biorisk Tacit Knowledge and Troubleshooting | accuracy | No clear effect |
| SFT on correct CTF solutions from a precursor of o3 (ablation) | gpt-oss-120b · CTFs | pass rate | No clear effect |
| internal best-of-k scaffolding at each step (ablation) | gpt-oss-120b · CTFs | pass rate: under four percentage points | No clear effect |
| bio pretraining data downsampling (gpt-oss vs o4-mini with same anti-refusal mix) (ablation) | gpt-oss · bio benchmarks | accuracy | No clear effect |
| use cyber MFT model vs original gpt-oss against OpenAI o3 | gpt-oss · cyber evals | pass rate | Hurt |
| MFT gpt-oss vs OpenAI o3 on biology evals | gpt-oss MFT · biology proxy evals | accuracy: one point better on Tacit Knowledge, worse on others | Mixed |
Closest papers by meaning
- Scaling Trends for Data Poisoning in LLMs (AAAI 2025 · similarity 0.86)
- Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs (NeurIPS 2026 · similarity 0.85)
- On Evaluating the Durability of Safeguards for Open-Weight LLMs (ICLR 2025 · similarity 0.84)
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.84)
- Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptation (ICML 2024 · similarity 0.83)
- The WMDP Benchmark: Measuring and Reducing Malicious Use with Unlearning (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
