Research map / Large language models / Language, safety and interpretability
Beyond Memorization: Violating Privacy via Inference with Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies whether pretrained LLMs can infer personal attributes from users' free text at inference time, builds the PersonalReddit labeled dataset, and tests anonymization, alignment, and adversarial chatbot threats.
Problem
Current LLM privacy research focuses on memorized training data extraction, while whether LLMs can infer personal attributes (location, income, sex, etc.) from user-written text at inference time, and whether mitigations stop this, is unexplored.
Approach
Formalizes privacy threats from LLM inference (free text inference and adversarial chatbots steering conversations), builds a human-labeled Reddit author-profiling dataset, and shows off-the-shelf LLMs infer attributes with high accuracy at far lower cost than human labelers, while anonymizers and alignment fail to prevent it.
Key results
- GPT-4 reaches 85.5% top-1 and 95.2% top-3 accuracy across attributes on PersonalReddit, at about 100x lower cost and 240x less time than human labelers.
- Anonymization with Azure PII removal reduces GPT-4 location accuracy only from about 86% to about 55%, and the accuracy drop shrinks to 7.7% at hardness 5.
- Simulated adversarial chatbots built on GPT-4 achieve 59.2% top-1 accuracy over 224 interactions, and Claude/Llama/PaLM refusals of privacy-invasive prompts are near zero (0% for Llama-2 and OpenAI).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase model size from Llama 2 7B to Llama 2 70B | Llama 2 · PersonalReddit, all attributes | top-1 accuracy: 51% -> 66% | Helped |
| Use top 3 guesses instead of the top guess | GPT-4 · PersonalReddit, all attributes | accuracy: 85.5% -> 95.2% | Helped |
| Anonymize text using AzureLanguageService (ablation) | GPT-4 · PersonalReddit, location | accuracy: ~86% -> ~55% | Hurt |
| Anonymize text using AzureLanguageService (ablation) | GPT-4 · PersonalReddit, hardness 1 | accuracy: 85.0% -> 43.9% | Hurt |
| Anonymize text using AzureLanguageService (ablation) | GPT-4 · PersonalReddit, hardness 2 | accuracy: 78.2% -> 59.2% | Hurt |
| Anonymize text using AzureLanguageService (ablation) | GPT-4 · PersonalReddit, hardness 3 | accuracy: 77.9% -> 64.2% | Hurt |
| Anonymize text using AzureLanguageService (ablation) | GPT-4 · PersonalReddit, hardness 4 | accuracy: 84.2% -> 52.6% | Hurt |
| Anonymize text using AzureLanguageService (ablation) | GPT-4 · PersonalReddit, hardness 5 | accuracy: 69.2% -> 61.5% | Hurt |
| Use current model alignment against privacy-invasive prompts | LLMs from tested providers · Privacy-invasive prompts | refusal rate: 0%; 0%; 2.8%; 10.7% | No clear effect |
| Use GPT-4 for zero-shot gender profiling rather than the best competition method | PAN 2018 English gender profiling · PAN 2018 | accuracy: 82.21% -> 90.2% | Helped |
Surprising findings
- Anonymization removes explicit location mentions yet GPT-4 still infers location with about 55% accuracy, indicating subtle contextual cues persist.
- Model providers refuse very few privacy-invasive prompts: 0% for Meta Llama-2 and OpenAI GPT, 2.8% for Anthropic Claude, and 10.7% for Google PaLM.
Limitations
- PersonalReddit is not publicly released due to personal data, so main results are hard to reproduce; only synthetic examples are shared.
- Many closed models accessed via API or Poe web interface, so model versions are partially uncontrolled.
- Adversarial chatbot results are simulated with bots rather than real users.
Experimental setup
Datasets: PersonalReddit (520 Reddit profiles, 5814 comments) ACSIncome New York 2018 PAN 2018 author profiling Enron-Email (decontamination context) PushShift Reddit
Benchmarks: PersonalReddit ACSIncome PAN 2018
Metrics: top-1 accuracy top-2 accuracy top-3 accuracy cost time accuracy per hardness level
Baselines: Human labelers GPT-3.5 Llama-2 7B Llama-2 13B Llama-2 70B PaLM 2 Text PaLM 2 Chat Claude-2 Claude-Instant XGBoost (ACSIncome) PAN 2018 top competition entry
Builds on: GDPR definitions of personal data ACS census datasets PAN author profiling competition
Closest papers by meaning
- A Synthetic Dataset for Personal Attribute Inference (NeurIPS 2024 · similarity 0.95)
- Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory (ICLR 2024 · similarity 0.92)
- Private Attribute Inference from Images with Vision-Language Models (NeurIPS 2024 · similarity 0.91)
- Language Models are Advanced Anonymizers (ICLR 2025 · similarity 0.91)
- Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.90)
- Self-Refining Language Model Anonymizers via Adversarial Distillation (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.