Research map / Large language models / Language, safety and interpretability

Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory

ICLR 2024 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 12 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ConfAIde, a four-tier benchmark grounded in contextual integrity theory that tests whether instruction-tuned LLMs reason about privacy-appropriate information flow, finding that even GPT-4 and ChatGPT leak secrets in interactive settings.

Problem

LLMs receiving information from multiple sources at inference time may expose sensitive input data in inappropriate contexts; existing privacy work focuses on training-data leakage rather than contextual information flow.

Approach

Grounds privacy evaluation in contextual integrity theory with four escalating tiers (information sensitivity out of context, information-flow expectations in context, theory-of-mind secret keeping, and meeting summarization/action-item leakage), measuring inference-time privacy leakage rather than memorization.

Key results

• GPT-4 and ChatGPT reveal secrets 22% and 93% of the time in Tier 3 (string-match leakage, worst case over prompts with privacy instructions).
• Correlation with human judgment drops for GPT-4 from 0.86 in Tier 1 to 0.10 in Tier 3.
• In Tier 4 meeting tasks, GPT-4 has 0.89 error rate for action items (leaks secret or omits public info) and chain-of-thought prompting makes leakage worse rather than better.

Measured effects reported in the paper

Surprising findings

• Chain-of-thought prompting ('Take a deep breath...') does not reduce leakage and often increases it.
• Removing explicit privacy-preserving instructions increases leakage, and models that are RLHF-tuned still leak secrets despite explicit instructions.

Limitations

• Tier 3 and Tier 4 scenarios are GPT-4 generated and may be familiar to GPT-4; some were filtered by annotators for safety and coherence.
• Leakage detection via exact string match may underestimate leakage; manual review found 16 missed leaks among 200 GPT-4 responses.
• Authors explicitly do not make normative judgments on what should be private, and human annotations have low inter-annotator agreement (Krippendorff's alpha 0.07-0.34).

Experimental setup

Datasets: Pew privacy study information types Martin & Nissenbaum vignette survey templates

Benchmarks: ConfAIde

Metrics: Pearson correlation with human judgments leakage rate (string match) leakage rate (proxy model) theory-of-mind error rate binary control question error rate omits public information rate

Baselines: ChatGPT InstructGPT Mixtral Llama-2 Chat Llama-2 Davinci

Builds on: contextual integrity factorial vignette survey (Martin & Nissenbaum) chain-of-thought prompting

Closest papers by meaning

• Position: Contextual Integrity is Inadequately Applied to Language Models (ICML 2025 · similarity 0.93)
• Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.92)
• Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.90)
• Contextual Integrity in LLMs via Reasoning and Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternatives (NeurIPS 2024 · similarity 0.90)
• Privacy Reasoning in Ambiguous Contexts (NeurIPS 2025 · similarity 0.90)