Research map / Large language models / Language, safety and interpretability

Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Controllable Safety Alignment (CSA), where LLMs follow natural-language safety configs in the system prompt, and CoSAlign, a data-centric SFT+DPO method, plus a CoSA-Score evaluation metric for controllability.

Problem

Safety alignment of LLMs is one-size-fits-all and static, ignoring cultural variation and users' differing safety needs, and re-aligning per user is prohibitively expensive.

Approach

Align the model to follow free-form natural-language safety configs placed in the system prompt, so safety behavior can be adjusted at inference time without retraining. Training data is built from harm-category-based synthetic configs, prompts from BeaverTails, and chosen/rejected responses from vanilla and refusal-removed GPT-4, then trained with SFT and DPO.

Key results

• CoSAlign reaches CoSA-Score 0.55 on seen configs and 0.47 on unseen configs, versus 0.29 for Llama3.1-8B-Instruct+ICA and 0.35 for GPT-4o-mini+ICA on seen configs.
• CoSAlign keeps default-config safety at 99.0% safe versus 98.3% for Llama3.1-8B-Instruct.
• MMLU drops only from 67.8 to 67.7 after CoSAlign training.

Measured effects reported in the paper

Surprising findings

• SFT-only CoSAlign has a low CoSA-Score (0.24) because it produces many helpful+unsafe responses, so preference learning is essential for controllability.
• GPT-4o-mini trained with instruction hierarchy shows poor safety controllability from system prompts.

Limitations

• Evaluation relies on synthetic configs and prompts built from predefined harm categories, and on GPT-4 as judge, with human verification only on a sample.
• Test sets are curated automatically; a human-curated real-world benchmark is still in progress.
• Only one base model (Llama3.1-8B-Instruct) is reported in the main results.

Experimental setup

Datasets: BeaverTails MMLU

Benchmarks: CoSA-Score test set (50 seen configs, 2000 prompts; 30 unseen configs, 1200 prompts) BeaverTails default-config safety (400 prompts) MMLU

Metrics: CoSA-Score helpful+safe rate helpful+unsafe rate percent safe

Baselines: in-context alignment (ICA) with Llama3.1-8B-Instruct Llama3-SFT + ICA refusal-removed Llama3.1-8B + ICA unsafe replacement (UR) GPT-4o-mini + ICA CoSAlign SFT-only

Builds on: direct preference optimization instruction hierarchy in-context alignment

Closest papers by meaning

• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.93)
• Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons (NeurIPS 2025 · similarity 0.92)
• Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.92)
• UpSafe℃: Upcycling for Controllable Safety in Large Language Models (NeurIPS 2026 · similarity 0.90)
• Safety Depth in Large Language Models: A Markov Chain Perspective (NeurIPS 2025 · similarity 0.90)
• RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards (NeurIPS 2025 · similarity 0.90)