Research map / Large language models / Instruction and fine-tuning

SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Safety alignment after fine-tuning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SEAL, a bilevel-optimization-learned data selector that down-ranks unsafe or low-quality fine-tuning samples so that LLM fine-tuning preserves safety alignment.

Problem

Fine-tuning aligned LLMs on task data, even benign data, can erode their safety alignment; the goal is to select fine-tuning data that preserves safety while maintaining downstream quality.

Approach

Learn a softmax data selector via a bilevel problem that minimizes the safety-data loss of a model fine-tuned on the reweighted fine-tuning data, solved with a penalty-based first-order gradient method and a memory-efficient variant that drops the auxiliary model update; then fine-tune on the top-p% ranked samples.

Key results

• On Llama-3-8b-Instruct, SEAL reaches 60.22 win rate on Anthropic HH and 69.29 on HEx-PHI versus 50 for standard SFT, with an average gain of about 8.5% over random selection.
• On Merlinite-7b, SEAL reaches 76.72 win rate on Anthropic HH and 55.78 on SlimOrca, with an average gain of about 9.7% over random selection.
• A data selector trained with the smaller Phi-3-mini model remains effective when fine-tuning Merlinite-7b, showing transferability across models.

Measured effects reported in the paper

Surprising findings

• On benign fine-tuning data with Llama2-7b-chat, DSIR and random selection do not improve safety meaningfully, and SafeInstr gave no advantage, in contrast to the harmful-data setting.
• On the Pythia-2.8b model, baselines (random, DSIR, SafeInstr) underperform standard SFT on the target domain, which the authors attribute to the weaker initial aligned model.

Limitations

• Data selector training adds substantial compute and memory overhead relative to standard fine-tuning.
• Performance gains depend on the quality of the initially aligned model; weaker models (Pythia-2.8b) show smaller gains.
• Evaluation relies on win rate judged by GPT-4 against an SFT reference rather than standard safety benchmarks.

Experimental setup

Datasets: RedOrca SlimOrca Anthropic HH HEx-PHI Alpaca-cleaned OpenOrca Anthropic red-teaming

Benchmarks: Anthropic HH test SlimOrca test HEx-PHI

Metrics: win rate vs standard SFT reference

Baselines: Standard SFT Random selection DSIR SafeInstr

Builds on: DSIR LESS DPO LoRA

Closest papers by meaning

• Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasets (ICML 2025 · similarity 0.89)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.89)
• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.89)
• Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models (ICML 2024 · similarity 0.88)
• SaLoRA: Safety-Alignment Preserved Low-Rank Adaptation (ICLR 2025 · similarity 0.88)
• Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.87)