Research map / Adversarial robustness and security / Attacks on text and graphs
Breach By A Thousand Leaks: Unsafe Information Leakage in 'Safe' AI Responses
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an information-theoretic threat model of inferential adversaries, an impermissible information leakage (IIL) metric, a question-decomposition attack, and a randomised-response information censorship defense with safety-utility trade-off bounds.
Problem
Safety evaluations and defenses for LLMs focus on whether single outputs are impermissible (jailbreak/ASR), missing risks from dual-intent subquestions whose composed answers leak dangerous knowledge.
Approach
Model adversaries as inferential (gaining information from victim outputs, measured by impermissible information leakage) rather than security (forcing explicitly impermissible outputs); instantiate with a Decomposition Attack that asks dual-use subquestions and aggregates answers, and show a randomised-response censorship mechanism bounds leakage at a utility cost.
Key results
- Decomposition Attack with Mistral-7B-Instruct adversary reached IIL 0.95±0.22 on WMDP-Bio versus 0.43±0.12 for PAIR.
- With Llama-8B-Instruct adversary, DA reached IIL 0.59 on WMDP-Bio and 0.38 on WMDP-Chem, versus 0.28 and 0.16 for PAIR.
- DA flagged far less often than PAIR for Llama-8B (0.03/0.11 vs 0.13/0.58 flags per attack on Bio/Chem).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DA instead of PAIR | Mistral-7B-Instruct adversary · WMDP-Bio | IIL: 0.43 ± 0.12 -> 0.95 ± 0.22 | Helped |
| Use DA instead of PAIR | Mistral-7B-Instruct adversary · WMDP-Chem | IIL: 0.32 ± 0.12 -> 0.48 ± 0.10 | Helped |
| Use DA instead of PAIR | Llama-8B-Instruct adversary · WMDP-Bio | IIL: 0.28 ± 0.06 -> 0.59 ± 0.09 | Helped |
| Use DA instead of PAIR | Llama-8B-Instruct adversary · WMDP-Chem | IIL: 0.16 ± 0.04 -> 0.38 ± 0.05 | Helped |
| Use DA instead of PAIR | Mistral-7B-Instruct adversary · WMDP-Bio | flags per attack: 1.15 -> 0.34 | Helped |
| Use DA instead of PAIR | Mistral-7B-Instruct adversary · WMDP-Chem | flags per attack: 2.07 -> 0.30 | Helped |
| Use DA instead of PAIR | Llama-8B-Instruct adversary · WMDP-Bio | flags per attack: 0.13 -> 0.03 | Helped |
| Use DA instead of PAIR | Llama-8B-Instruct adversary · WMDP-Chem | flags per attack: 0.58 -> 0.11 | Helped |
Surprising findings
- Safety-aligned Llama-3.1-8B-Instruct as adversary was very effective at remaining undetected by filters.
Limitations
- Evaluation uses small curated subsets (35 Bio and 64 Chem questions) and multiple-choice proxies.
- Randomised response defense requires a safety set and bounds leakage at a utility cost, which the authors note is inevitable.
- Impermissibility is defined by the authors via Llama-Guard flags rather than ground-truth harm assessment.
Experimental setup
Datasets: WMDP-Bio WMDP-Chem
Benchmarks: WMDP-Bio WMDP-Chem
Metrics: impermissible information leakage (IIL) filter flag rate two-sample t-test p-values
Baselines: PAIR
Builds on: PAIR WMDP Llama-Guard-3-8B Randomised response
Closest papers by meaning
- Bits Leaked per Query: Information-Theoretic Bounds for Adversarial Attacks on LLMs (NeurIPS 2025 · similarity 0.90)
- Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge (NeurIPS 2026 · similarity 0.89)
- Safety Alignment Can Be Not Superficial With Explicit Safety Signals (ICML 2025 · similarity 0.89)
- Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Models (ICLR 2025 · similarity 0.88)
- Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs (NeurIPS 2026 · similarity 0.88)
- Inadvertent Context Leakage in Language Models (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.