Research map / Adversarial robustness and security / Attacks on text and graphs
Bits Leaked per Query: Information-Theoretic Bounds for Adversarial Attacks on LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an information-theoretic framework bounding the number of adversarial queries against LLMs by the mutual information (bits leaked per query) between the observable signal and the attack target, and validates the inverse law empirically on seven LLMs across three attack types.
Problem
Adversarial attacks on LLMs are evaluated empirically by success rate versus query count, with no principled measure of how much observable signal (answer tokens, thinking process, logits) leaks and how many queries an attacker minimally needs.
Approach
Treats the mutual information I(Z;T) between the model's observable signal and the attack target property as leaked bits per query, and derives a lower bound N_min >= log(1/eps)/I(Z;T) via Fano's inequality, with a matching SPRT-based upper bound; experiments check the predicted log-log slope of -1.
Key results
- Adaptive attacks yield log-log regression slopes between -0.97 and -1.03 across seven models, consistent with the predicted N proportional to 1/I.
- Exposing answer tokens alone requires about a thousand queries; adding logits cuts this to about a hundred; revealing the full thinking process trims it to a few dozen.
- Non-adaptive attacks deviate strongly from the -1 slope with p < 10^-3 for all models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive queries that update from intermediate feedback rather than a fixed non-adaptive query policy | LLM attacks across seven models and three attack tasks · System-prompt leakage, jailbreak, and relearning | log-log slope of query count versus mutual information: Adaptive slopes remain close to the theoretical value -1; non-adaptive slopes deviate strongly | Helped |
| Reveal logits in addition to answer tokens | LLM attacks · System-prompt leakage, jailbreak, and relearning | queries to attack success: about a thousand queries; adding logits cuts this to about a hundred | Helped |
| Reveal the full thinking process in addition to answer tokens | LLM attacks · System-prompt leakage, jailbreak, and relearning | queries to attack success: revealing the full thinking process trims it to a few dozen | Helped |
| Expose logits or thinking-process tokens | LLM attacks · System-prompt leakage, jailbreak, and relearning | queries to attack success: reduces the query budget by roughly one order of magnitude | Helped |
| Reveal reasoning through thinking-process tokens or logits | LLM attacks · Jailbreak | median query budget: increases leakage by approximately 0.5 bit per query; cuts the median jailbreak budget from thousands of queries to tens | Helped |
| Lower temperature or nucleus threshold to reduce output entropy (ablation) | LLM attacks · System-prompt leakage, jailbreak, and relearning | queries to attack success | Hurt |
| Reveal additional signals such as logits or thinking-process tokens (ablation) | LLM attacks · System-prompt leakage, jailbreak, and relearning | queries to attack success: just a few extra leaked bits cut the query budget by orders of magnitude | Helped |
| Use non-adaptive attacks instead of adaptive attacks | LLM attacks · System-prompt leakage, jailbreak, and relearning | log-log slope of query count versus mutual information: Adaptive slopes remain close to the theoretical value -1; non-adaptive slopes deviate strongly | Hurt |
Surprising findings
- Non-adaptive attacks deviate substantially from the predicted inverse law, with slopes between -0.32 and 0.26.
Limitations
- Non-adaptive attacks do not follow the predicted inverse scaling, so the bound's tightness depends on attacker adaptivity.
- Mutual information is estimated with variational lower bounds using a RoBERTa critic, so estimates are approximate.
- The lower bound is stated for i.i.d. leakage across queries, an assumption that may not hold in practice.
Experimental setup
Datasets: system-prompt-leakage AdvBench Dolma Wikibooks
Benchmarks: system-prompt-leakage AdvBench Dolma Wikibooks relearning
Metrics: queries to success N mutual information I(Z;T) log-log regression slope ROUGE-L OpenAI Moderation flag
Baselines: GCG PAIR paraphrase rewriting
Builds on: Fano's inequality sequential probability ratio test MINE InfoNCE greedy coordinate gradient PAIR
Closest papers by meaning
- Breach By A Thousand Leaks: Unsafe Information Leakage in 'Safe' AI Responses (ICLR 2025 · similarity 0.90)
- Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space (NeurIPS 2024 · similarity 0.90)
- Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge (NeurIPS 2026 · similarity 0.90)
- Inadvertent Context Leakage in Language Models (NeurIPS 2026 · similarity 0.89)
- Exploiting LLM Quantization (NeurIPS 2024 · similarity 0.89)
- Protecting Your LLMs with Information Bottleneck (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.