Research map / Large language models / Language, safety and interpretability
A Synthetic Dataset for Personal Attribute Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds an LLM-agent Reddit-style comment-thread simulator seeded with synthetic personal profiles and releases SynthPAI, a ~7.8K-comment synthetic dataset with human-verified personal attribute labels for privacy-preserving attribute inference research.
Problem
Personal attribute inference (PAI) from online text by LLMs is a privacy threat, but research is hampered by a lack of public, realistic datasets because real personal data cannot be released for ethical and legal reasons.
Approach
Simulate multi-party Reddit comment threads with LLM agents seeded by synthetic personal profiles and writing styles, auto-tag inferrable attributes with an LLM oracle, then manually verify labels to produce a privacy-free PAI dataset that reproduces conclusions from real-data studies.
Key results
- Human participants (40) distinguishing synthetic from real Reddit comments reached only 51.9% accuracy, barely above chance.
- Across 18 LLMs on SynthPAI, GPT-4 achieved the highest attribute inference accuracy (76%), closely followed by Claude-3 Opus, with consistent rankings matching real-data findings.
- SynthPAI contains 7823 comments over 103 threads and 300 profiles, with 1110 human-verified profile-level labels across 8 attributes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use SynthPAI comments instead of real Reddit comments for human real-versus-generated classification | Human participants · SynthPAI comments vs. real Reddit comments | human classification accuracy: 51.9% | No clear effect |
| evaluate personal attribute inference on SynthPAI rather than the real-world dataset | 18 state-of-the-art LLMs · PAI across attributes | accuracy: drop of $\sim 10\%$ in overall accuracy | Hurt |
| increase model size | LLM families evaluated for PAI · SynthPAI | PAI performance | Helped |
| use newer model iterations | Llama-3 and Claude-3 · SynthPAI | PAI performance | Helped |
| compare GPT-4 accuracy on SynthPAI with reported real-world results | GPT-4 · PAI across attributes | accuracy: +2.3; -5; +5.2; -12.3; -6.2; -4.2; -4.7; -8.9 | Mixed |
| anonymize comments with Azure Language Services | GPT-4 · SynthPAI, anonymized comments, location | accuracy: around $50\%$ accuracy | Hurt |
| use human rather than LLM-generated profile-level labels | SynthPAI label aggregation · SynthPAI profile-level labels | agreement: $76\%$ to $83\%$ | Helped |
Surprising findings
- Even explicitly removed attributes such as location remain inferable after Azure anonymization, with GPT-4 retaining around 50% accuracy on location.
- LLM tags had a 0.14 false negative rate but only 0.01 false positive rate against human labels, so model-tagged leaks are almost always confirmed by humans.
Limitations
- Model-provided tags are only around 80% accurate, requiring manual correction.
- Dataset is limited to a thread-based, English comment setting and eight attributes.
- Synthetic comments are slightly harder than real ones since humans rarely state obvious attributes, causing about a 10% accuracy drop.
Experimental setup
Datasets: PersonalReddit
Benchmarks: SynthPAI
Metrics: accuracy human discrimination accuracy label agreement (false negative and false positive rates)
Baselines: Claude-3 Opus Llama-3 Human study (real Reddit comments)
Builds on: GPT-4 synthetic profile generation Staab et al. beyond memorization PAI evaluation
Closest papers by meaning
- Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.95)
- The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text (ICML 2025 · similarity 0.90)
- Language Models are Advanced Anonymizers (ICLR 2025 · similarity 0.89)
- Private Attribute Inference from Images with Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory (ICLR 2024 · similarity 0.88)
- PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.