Research map / Large language models / Instruction and fine-tuning

On Evaluating the Durability of Safeguards for Open-Weight LLMs

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Safety alignment after fine-tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Re-evaluates two durable safeguards for open-weight LLMs (RepNoise and TAR) and shows that seeds, trainer implementation, fine-tuning configs, prompt templates, and benchmark choice can make the defenses appear far less durable than originally claimed.

Problem

Durable safeguards for open-weight LLMs, meant to resist weight modification via fine-tuning, are claimed to hold under adversarial fine-tuning, but the reliability of such durability evaluations is unclear.

Approach

Case-study re-evaluation of RepNoise and TAR showing that small, plausible evaluation choices (random seeds, trainer implementation, LR warmup/decay, LoRA, prompt template, benign fine-tuning, side-effect benchmarks) can overturn durability claims, and distilling these into evaluation pitfalls and a checklist.

Key results

• RepNoise at learning rates 6e-5 and 8e-5 reached harmfulness across 5 seeds nearly equal to undefended Llama-2-7B-Chat, breaking the claimed defense by rerunning the same attack with different seeds.
• TAR-Bio-v2 post-attack WMDP-Bio accuracy exceeded 60% across three original fine-tuning configs over 5 seeds, versus the original paper's near-random or ~45% reports; New-Config 1 (100 warmup steps, cosine decay) reliably recovers accuracy.
• TAR-Bio-v2 showed GPT-2 perplexity of 16.4 on BenignBio versus 3.8 for the undefended model, and TAR checkpoints dropped to near zero on GSM8K (1.5 for v1, 4.8 for v2).

Measured effects reported in the paper

Surprising findings

• Fine-tuning on 100 benign data points or on the retain set alone largely recovers supposedly unlearned harmful capabilities in both RepNoise and TAR.
• TAR-Bio-v1 accuracy on WMDP-Bio rose from near-random (~25%) to above 45% under a chat-template prompt change with answer extraction, indicating the unlearning was overestimated.

Limitations

• Study limited to two case-study defenses (RepNoise and TAR) and mostly the biosecurity TAR checkpoint.
• Cannot enumerate all possible fine-tuning configurations or prompt templates, so coverage is incomplete.

Experimental setup

Datasets: BeaverTails Pile-Bio Forget Pile-Bio Retain WMDP HarmfulQA HEx-PHI SORRY-bench AOA Alpaca MMLU GSM8K MATH

Benchmarks: BeaverTails WMDP SORRY-bench HEx-PHI MMLU GSM8K MATH BBH HumanEval MT-Bench TruthfulQA BenignBio

Metrics: harmfulness score harmfulness rate WMDP-Bio accuracy accuracy GPT-2 perplexity worst-case attack success over random seeds

Baselines: RepNoise TAR-Bio-v1 TAR-Bio-v2 Llama-2-7B-Chat Llama-3-8B-Instruct

Builds on: Representation Noising Tamper Attack Resistance RMU WMDP

Closest papers by meaning

• Tamper-Resistant Safeguards for Open-Weight LLMs (ICLR 2025 · similarity 0.92)
• On Calibration of LLM-based Guard Models for Reliable Content Moderation (ICLR 2025 · similarity 0.90)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.90)
• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.90)
• Representation Noising: A Defence Mechanism Against Harmful Finetuning (NeurIPS 2024 · similarity 0.89)
• MoGU: A Framework for Enhancing Safety of LLMs While Preserving Their Usability (NeurIPS 2024 · similarity 0.89)