Research map / Large language models / Language, safety and interpretability

Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models

ICLR 2025 · Large language models › Language, safety and interpretability · approach: Jailbreak attacks and defenses · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Jailbreak Antidote, which shifts a sparse top-5% subset of last-token hidden states along a PCA-derived safety direction at inference to trade off safety and utility without token overhead.

Problem

Jailbreak defenses for LLMs add inference overhead (extra tokens, prompts) or require costly safety fine-tuning, lack runtime flexibility, and often over-refuse benign queries, degrading utility.

Approach

Compute a safety direction as the top principal component of last-token hidden states from benign vs. harmful prompts, keep only the top-k% dimensions by magnitude, and add the scaled masked direction to hidden states at inference; the scale alpha tunes the safety-utility trade-off at runtime.

Key results

• On Llama-3-70B-it, Jailbreak Antidote reaches 100% DSR on JailbreakBench while keeping a 53.5% AlpacaEval win rate versus the undefended model.
• On Llama-3-8B-it, DSR rises from 68.9% (baseline) to 99.4%, with win rate 53.0%.
• Adjusting top 5% of dimensions is reported as nearly as effective as adjusting 100%, with far less utility loss.

Measured effects reported in the paper

Surprising findings

• Negative alpha reverses the method, lowering safety below baseline, so it can act as an attack.
• Win rate drops sharply when all dimensions (k=100%) are adjusted.

Limitations

• Negative alpha can be exploited by attackers to weaken safety.
• Smaller models show lower DSR than larger ones, suggesting less capacity to encode safety information.
• Alpha range chosen per model by coherence bounds, requiring per-model tuning.

Experimental setup

Datasets: JailbreakBench Safety-Prompts dataset

Benchmarks: JailbreakBench AlpacaEval

Metrics: Defense Success Rate Win Rate Length Controlled Win Rate runtime per query

Baselines: In-Context Learning Paraphrase Perplexity Filter Self-Reminder SemanticSmoothLLM SmoothLLM Llama-3-8B-it-RR Gemma-2-9B-it-DSA

Builds on: representation engineering PAIR GCG SmoothLLM

Closest papers by meaning

• UpSafe℃: Upcycling for Controllable Safety in Large Language Models (NeurIPS 2026 · similarity 0.92)
• STAIR: Improving Safety Alignment with Introspective Reasoning (ICML 2025 · similarity 0.91)
• Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment (CVPR 2025 · similarity 0.91)
• SafePTR: Token-Level Jailbreak Defense in Multimodal LLMs via Prune-then-Restore Mechanism (NeurIPS 2025 · similarity 0.91)
• RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards (NeurIPS 2025 · similarity 0.90)
• On Prompt-Driven Safeguarding for Large Language Models (ICML 2024 · similarity 0.90)