Research map / Large language models / Language, safety and interpretability

Protecting Your LLMs with Information Bottleneck

NeurIPS 2024 · Large language models › Language, safety and interpretability · approach: Jailbreak elicitation and risk · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes IBProtector, a lightweight trainable extractor guided by an information-bottleneck objective that masks and perturbs jailbreak prompts so a frozen target LLM refuses harmful requests while still answering benign ones.

Problem

LLMs remain vulnerable to token-level (GCG) and prompt-level (PAIR) jailbreaks; existing defenses rely on random perturbations, costly semantic parsing, or fine-tuning of the target model.

Approach

Applies the information bottleneck principle to jailbreak defense: a small trainable extractor (TinyLlama-1.1B with an MLP head) produces Bernoulli masks that keep only response-relevant prompt tokens and replace the rest with a period, trained with a KL-based compactness bound, continuity loss, and an informativeness loss computed through the frozen target LLM; an RL variant is proposed for black

Key results

• On PAIR attacks against Vicuna-13B, IBProtector reduces ASR from 87.5% to 19.2%, and on GCG attacks against LLaMA-2-7b-chat from 27.5% to 0.8% (original attack vs. IBProtector).
• IBProtector keeps benign answering rate high on TriviaQA (96.5% on Vicuna-13B vs. 97.8% undefended).
• On EasyJailbreak transfer attacks against Vicuna-13B, ASR drops from 88.6% to 18.9% without retraining on those attacks.

Measured effects reported in the paper

Surprising findings

• Smooth-style random perturbation defenses nearly lose their effect or even increase ASR on stealthier EasyJailbreak attacks, while the learned extractor transfers.

Limitations

• Requires shared tokenizer and gradient access to the target LLM, so it is not directly applicable to closed models like GPT-4; RL variant is proposed but not tested.
• Extracted sub-prompts are non-fluent and hard for humans to interpret; defense relies on the target model's own safety alignment.
• Evaluated only on text and small-scale attack sets (e.g., 120 test instances, 50 adaptive instances).

Experimental setup

Datasets: AdvBench TriviaQA EasyJailbreak

Benchmarks: AdvBench TriviaQA EasyJailbreak

Metrics: attack success rate harm score GPT-4 score benign answering rate

Baselines: Fine-tuning Unlearning LLM Self Defense Smooth LLM RA-LLM Semantic Smooth

Builds on: information bottleneck variational information bottleneck SmoothLLM GCG PAIR

Closest papers by meaning

• RigorLLM: Resilient Guardrails for Large Language Models against Undesired Content (ICML 2024 · similarity 0.89)
• Bits Leaked per Query: Information-Theoretic Bounds for Adversarial Attacks on LLMs (NeurIPS 2025 · similarity 0.89)
• Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge (NeurIPS 2026 · similarity 0.88)
• Look Before You Leap: Enhance Attention and Vigilance Regarding Harmful Content with GuidelineLLM (AAAI 2025 · similarity 0.88)
• Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space (NeurIPS 2024 · similarity 0.87)
• Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models (ICLR 2025 · similarity 0.87)