Research map / Adversarial robustness and security / Attacks on text and graphs

Inadvertent Context Leakage in Language Models

NeurIPS 2026 · Adversarial robustness and security › Attacks on text and graphs · approach: LLM jailbreak and injection defense · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that secrets in an LLM's context leak into benign outputs via model-specific statistical channels, letting adaptive decoders recover numeric secrets, user-memory attributes, and a full SSN via an RL-optimized prompt-injection attack.

Problem

LLM agents hold sensitive in-context secrets; linguistic judges only check whether the secret is stated, so secrets may still leak through statistical properties of benign outputs even when the model refuses direct extraction.

Approach

Formalize inadvertent context leakage as a predicate inference game where an adaptive adversary, using black-box queries to a model, trains a decoder (on self-authored contexts) to read model-specific channels in benign outputs; also introduces "suppression" (avoidance of the protected value) as the mechanism linking leakage to output distortion.

Key results

• Adaptive adversary reaches 100% full-secret reconstruction at N=2 on Claude Opus 4.6 and Gemini 3.1 Pro, and 82% exact match at N=4 on Claude Opus 4.6 (vs 44% for static templates).
• On CIMemories, the trained Llama-3.2-3B adversary reaches 0.319 advantage vs 0.058 for the DeepSeek-R1 linguistic judge against Claude Opus 4.6.
• RL-optimized prompt injection recovers the leading SSN digit in 97.1% of trials on Claude Opus 4.6 and 88.6% on Gemini 3.1 Pro; direct requests never elicit the SSN.

Surprising findings

• Stronger confidentiality instructions increased leakage on Claude Opus 4.6 (per-digit accuracy 0.643 under verbose policy vs 0.054 under terse), suggesting suppression drives the channel.
• Digit-stripped decoders still recover significant information from benign outputs, indicating non-literal channels.

Limitations

• Experiments focus on digit secrets and binary memory attributes on a fixed set of proprietary models.
• Estimation requires the adversary to identify the target model and pay a one-time querying cost; decoders are trained per model.
• Secrets are uniformly random; real secrets with structure may behave differently.

Experimental setup

Datasets: WildChat-1M CIMemories

Benchmarks: digit reconstruction (N=1,2,4,8) CIMemories memory predicates SSN extraction from personal agent WildChat single-digit recovery

Metrics: full-secret exact match per-digit accuracy channel entropy advantage (TPR-FPR) TPR at baseline FPR suppression Spearman correlation

Baselines: majority vote frequency-based Bayesian decoder DeepSeek-R1 linguistic judge

Builds on: CIMemories GRPO LoRA

Closest papers by meaning

• Bits Leaked per Query: Information-Theoretic Bounds for Adversarial Attacks on LLMs (NeurIPS 2025 · similarity 0.89)
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory (ICLR 2024 · similarity 0.88)
• Eliciting Secret Knowledge from Language Models (NeurIPS 2026 · similarity 0.88)
• Breach By A Thousand Leaks: Unsafe Information Leakage in 'Safe' AI Responses (ICLR 2025 · similarity 0.87)
• SentinelLMs: Encrypted Input Adaptation and Fine-Tuning of Language Models for Private and Secure Inference (AAAI 2024 · similarity 0.86)
• Measuring Non-Adversarial Reproduction of Training Data in Large Language Models (ICLR 2025 · similarity 0.86)