Research map / Large language models / Language, safety and interpretability

Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Jailbreak elicitation and risk · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Discovers a "safety basin" in LLM weight space, where random weight perturbations preserve safety until a sharp drop, and proposes the VISAGE safety metric (average safety margin over random perturbation directions) to measure finetuning risk.

Problem

Finetuning with a few adversarial or even benign examples can compromise LLM safety alignment; there is no principled, task-agnostic way to measure how risky finetuning is for a given aligned model.

Approach

Visualize the LLM safety landscape by perturbing weights along random and interpolated directions, revealing a flat "safety basin" around the aligned model that sharply collapses outside; define VISAGE as the average safety margin over sampled perturbations, and use checkpoint trajectories projected onto the landscape to explain how finetuning drags models out of the basin.

Key results

• Safety basin observed universally across LLaMA2-7B-chat, LLaMA3-8B-instruct, Mistral-7B-instruct-v0.2 and Vicuna-7B-v1.5: random perturbations preserve safety locally, then ASR jumps sharply outside the basin.
• Finetuning LLaMA2-7B-chat on 100 harmful examples raises AdvBench-520 ASR from 0.2 to 95.4, whereas mixing 100 harmful with 100 safe examples keeps the model safe (ASR 0.2 on 520-prompt set for the mix setting in the table).
• VISAGE ranks LLaMA2-7B-chat (85.32) above Vicuna-7B-v1.5 (73.26), matching the finetuning ASR ordering; removing the system prompt drops LLaMA2's VISAGE by 4.64 points and Mistral-7B-instruct-v0.1's by 53.33 points.

Measured effects reported in the paper

Surprising findings

• Removing the default system prompt reveals that the original Vicuna model is a local maximum of safety, implying some nearby perturbed weights are safer than the fine-tuned model.
• Jailbreak prompts are highly sensitive to weight perturbations, so slight perturbations can significantly lower ASR, suggesting a naive perturbation-based defense.

Limitations

• VISAGE is an average-depth proxy; basin width and smoothness are not yet captured by the metric.
• Evaluation relies mainly on refusal keyword detection and AdvBench, and the paper states Adv 520 ASR for the aligned models is already low (close across models), limiting discrimination.
• Analysis covers a small set of open-source 7B-class models plus LLaMA2-13B, and only full-parameter finetuning on one harmful dataset.

Experimental setup

Datasets: Anthropic red-teaming dataset (harmful samples) AdvBench MT-Bench MMLU (abstract_algebra, high_school_us_history, us_foreign_policy) POSE benchmark JailbreakBench

Benchmarks: AdvBench Harmful Behaviors (80 and 520 prompts) JailbreakBench POSE MMLU MT-Bench

Metrics: attack success rate (refusal keyword detection) VISAGE score perplexity LLaMA Guard 2 unsafe rate

Baselines: LLM Judge Llama Guard 2

Builds on: AdvBench GCG PAIR Safety prompt optimization (prompt tuning for safety) Llama Guard

Closest papers by meaning

• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.95)
• Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons (NeurIPS 2025 · similarity 0.93)
• Safety Depth in Large Language Models: A Markov Chain Perspective (NeurIPS 2025 · similarity 0.93)
• Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.93)
• Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.93)
• Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements (ICLR 2025 · similarity 0.93)