Research map / Large language models / Language, safety and interpretability

The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text

ICML 2025 · Large language models › Language, safety and interpretability · approach: Data and hallucination detection

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Audits membership inference risk of LLM-generated synthetic text using data-only attacks (n-gram and similarity-based) and proposes canaries with in-distribution prefix and high-perplexity suffix to better expose leakage.

Problem

How much information about fine-tuning records leaks through synthetic text generated by LLMs when the adversary only sees the synthetic data, and how should canaries be designed to audit this risk?

Approach

Data-based membership inference that uses only synthetic samples (via n-gram model fitted on synthetic data or similarity to nearest synthetic records) combined with RMIA scoring, plus canaries with an in-distribution low-perplexity prefix and high-perplexity suffix that leave detectable traces in generated text.

Key results

• Data-only 2-gram MIA reaches AUC 0.74 on SST-2, 0.68 on AG News and 0.77 on SNLI (standard canaries).
• Data-based attack needs canaries repeated roughly 8-16x more to match model-based attack vulnerability.
• With DP-SGD at epsilon=8, the strongest data-based MIA drops to AUC about 0.48-0.52 while SST-2 synthetic utility stays at 91.6% accuracy.

Measured effects reported in the paper

Surprising findings

• High-perplexity canaries that are highly vulnerable to model-based attacks become less detectable in synthetic data, with data-based AUC falling toward 0.5 at high perplexity.
• Canaries with an in-distribution prefix and high-perplexity suffix outperform fully in-distribution or fully out-of-distribution canaries for data-based attacks.

Limitations

• Auditing uses worst-case settings with 4 reference models trained on the full private data and known training procedure, which is optimistic for practical attackers.
• Data-based attacks rely on synthetic data volume and only a single model (Mistral-7B) and three short-text datasets were studied.

Experimental setup

Datasets: SST-2 AG News SNLI

Benchmarks: SST-2 AG News SNLI

Metrics: ROC AUC TPR at low FPR

Baselines: RMIA model-based attack Jaccard similarity attack embedding similarity attack

Builds on: RMIA LoRA DP-SGD Opacus

Closest papers by meaning

• Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.91)
• A Synthetic Dataset for Personal Attribute Inference (NeurIPS 2024 · similarity 0.90)
• SynBench: A Benchmark for Differentially Private Text Generation (NeurIPS 2026 · similarity 0.90)
• Privacy Amplification Through Synthetic Data: Insights from Linear Regression (ICML 2025 · similarity 0.89)
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory (ICLR 2024 · similarity 0.89)
• PrivAuditor: Benchmarking Data Protection Vulnerabilities in LLM Adaptation Techniques (NeurIPS 2024 · similarity 0.89)