Research map / Large language models / Language, safety and interpretability

On Prompt-Driven Safeguarding for Large Language Models

ICML 2024 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes how safety prompts shift LLM hidden representations toward a "refusal direction" and proposes DRO, which optimizes continuous safety prompt embeddings to move harmful/harmless query representations along/opposite that direction.

Problem

Safety prompts prepended to LLM inputs are a common lightweight safeguard, but their working mechanisms are not understood, and their effectiveness varies across prompts and models, limiting automatic optimization.

Approach

Analyze safety prompts in representation space (PCA on last-token hidden states) and find they mainly shift queries toward a "higher-refusal" direction rather than improving harmfulness recognition; then optimize continuous safety prompt embeddings with a loss on a logistic-regression refusal direction in a low-dimensional subspace, plus a regularizer keeping the orthogonal complement near the ori

Key results

• DRO reduces average compliance on AdvBench from 10.3% (default safety prompt) to 1.4%, and on MaliciousInstruct from 9.8% to 1.6%.
• DRO keeps AlpacaEval average win rate at 63.5 vs 62.5 for the default prompt, while vPT drops to 56.3.
• Under GCG jailbreak on AdvBench, DRO reduces average compliance from 13.3% to 3.1%.

Measured effects reported in the paper

Surprising findings

• Safety prompts do not clearly improve models' ability to distinguish harmful from harmless queries in representation space; instead they shift representations toward a higher-refusal direction.
• Even the basic safety prompt gives non-trivial protection under GCG jailbreak (54.1% to 13.3% compliance).

Limitations

• Queries are unambiguously harmful or harmless; ambiguous or persuasive adversarial queries are not addressed.
• Safety prompts alone are insufficient and need safety training; evaluation limited to 7B models.
• Harmfulness judging relies on LlamaGuard and string matching, which may be imperfect.

Experimental setup

Datasets: synthetic harmful and harmless 'How to' queries (100+100) generated with gpt-3.5-turbo AdvBench

Benchmarks: MaliciousInstruct AdvBench AlpacaEval held-out harmless query set AdvBench with GCG jailbreak

Metrics: % compliance on harmful queries % refusal on harmless queries win rate on AlpacaEval

Baselines: no prompt default safety prompt mistral safety prompt short safety prompt vanilla Prompt-Tuning (vPT)

Builds on: prompt tuning vanilla prompt-tuning AlpacaEval GCG

Closest papers by meaning

• Safety Pretraining: Toward the Next Generation of Safe AI (NeurIPS 2025 · similarity 0.91)
• Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.90)
• Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models (ICLR 2025 · similarity 0.90)
• Safety Reconstructed: Generative Modeling via Masked Diffusion Builds Strong Safety Guardrails (NeurIPS 2026 · similarity 0.90)
• Do as I do (Safely): Mitigating Task-Specific Fine-tuning Risks in Large Language Models (ICLR 2025 · similarity 0.90)
• RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards (NeurIPS 2025 · similarity 0.89)