Research map / Large language models / Language, safety and interpretability

Learning Safety Constraints for Large Language Models

ICML 2025 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SaP, which learns a polytope of safety constraints in an LLM's representation space via a concept encoder and CPM-style loss, and steers activations back into the safe region at inference to defend against jailbreaks without weight updates.

Problem

LLMs produce harmful outputs and are vulnerable to adversarial jailbreak attacks; existing defenses are brittle prompt tricks, costly retraining (safe RLHF), or lack interpretability and post-hoc correction.

Approach

Model the safe set as a learned polytope (intersection of halfspaces) in the LLM's intermediate representation space, trained from binary safe/unsafe labels using a Convex Polytope Machine loss with a concept encoder and sparsity regularization; unsafe activations are steered back into the polytope via Lagrangian-relaxed optimization at inference time.

Key results

• On Llama2-7B, SaP reduces average ASR from 12.92% to 0.26% while MMLU stays at 45.7% vs 45.8% original.
• On Ministral-8B, ASR drops from 55.77% to 3.25% with MMLU 63.3% vs 63.4%.
• On Qwen2-1.5B, ASR falls from 27.57% to 11.81% with MMLU 52.7% unchanged.

Measured effects reported in the paper

Surprising findings

• Polytope test classification accuracy is sometimes lower than MLP, yet SaP gives much better defense, showing classification accuracy alone does not predict defense capability.
• Many polytope facets remain inactive during training, suggesting inefficient use of facet capacity under the CPM loss.

Limitations

• Ministral-8B under many attacks produces semantically incoherent outputs after steering.
• Theoretical guarantees (PAC-style sample complexity) rely on strong assumptions hard to verify for LLMs.
• The CPM facet assignment is heuristic and many facets remain inactive.

Experimental setup

Datasets: HarmBench BeaverTails JailBreakBench

Benchmarks: HarmBench JailBreakBench MMLU MT-Bench BeaverTails

Metrics: attack success rate MMLU accuracy classification accuracy mutual information MT-Bench score

Baselines: Self Reminder Response Check SmoothLLM Rejection Sampling MLP MLP+5 MCA Safe-RLHF

Builds on: Convex Polytope Machine constrained Markov decision processes sparse autoencoders SafeFlow

Closest papers by meaning

• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.92)
• Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.91)
• Safety Pretraining: Toward the Next Generation of Safe AI (NeurIPS 2025 · similarity 0.89)
• Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.89)
• On Prompt-Driven Safeguarding for Large Language Models (ICML 2024 · similarity 0.89)
• UpSafe℃: Upcycling for Controllable Safety in Large Language Models (NeurIPS 2026 · similarity 0.89)