Research map / Large language models / Language, safety and interpretability

Language Models are Advanced Anonymizers

ICLR 2025 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes evaluating text anonymization against a strong LLM adversary and a feedback-guided adversarial anonymization framework where an inference LLM guides an anonymizer LLM over several rounds, outperforming Azure Language Service on PersonalReddit.

Problem

Existing text anonymizers (e.g., Azure, Presidio) mask explicit identifiers and fail to remove contextual cues that LLMs can use to infer personal attributes, and span-based metrics do not measure privacy against such inference.

Approach

Evaluate anonymization by whether a strong adversarial LLM can still infer personal attributes from the output, and anonymize iteratively: an adversary LLM infers attributes from the current text and an anonymizer LLM rewrites the relevant spans until inference fails or a round limit is reached.

Key results

• GPT-4-AA after five rounds reduces adversarial inference accuracy to 41.6% (absolute 24.7-point improvement over Azure's 66.3%) while keeping higher mean utility than Azure.
• In a human study (n=50), adversarial anonymization raised readability from 0.38 (Azure) to 0.89 and usability from 0.37 to 0.93 (normalized mean-over-median).
• On city-level location, GPT-4-AA reduced adversarial accuracy from Azure's 24.6% to 6.8% after one round and 1.3% after three rounds.

Measured effects reported in the paper

Surprising findings

• Even after three rounds, about 45% of location labels remain inferable by the adversary, mostly at country level.
• Azure's mean utility of 0.76 barely changes privacy, dropping accuracy only from 86.7% to 66.3%.

Limitations

• Privacy is bounded by the strength of the adversary and provides no formal guarantee against stronger adversaries with background knowledge.
• The real-world PersonalReddit dataset cannot be released for privacy reasons, limiting reproducibility on real data.
• Experiments are on 426 profiles restricted to under 1000 tokens; the LLM utility judge is GPT-4, which may be biased toward its own outputs.

Experimental setup

Datasets: PersonalReddit SynthPAI synthetic PersonalReddit conversations (Staab et al.)

Benchmarks: PersonalReddit SynthPAI synthetic PersonalReddit conversations

Metrics: adversarial accuracy readability meaning preservation BLEU ROUGE-1 human usability/typicality ratings

Baselines: Azure Language Service Dou-SD Dipper-XXL GPT-4 Base (no adversarial feedback)

Builds on: Staab et al. personal attribute inference with LLMs (PersonalReddit) Presidio / Azure Language Service Dipper paraphrasing Dou et al. self-disclosure detection

Closest papers by meaning

• Self-Refining Language Model Anonymizers via Adversarial Distillation (NeurIPS 2025 · similarity 0.94)
• Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.91)
• A Synthetic Dataset for Personal Attribute Inference (NeurIPS 2024 · similarity 0.89)
• Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.88)
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory (ICLR 2024 · similarity 0.88)
• The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text (ICML 2025 · similarity 0.88)