Research map / Large language models / Language, safety and interpretability

Unlocking Post-hoc Dataset Inference with Synthetic Data

ICML 2025 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes generating synthetic held-out data via a LoRA-finetuned Llama 3 8B suffix-completion generator plus a dual-classifier post-hoc calibration with a t-test, so dataset inference works without in-distribution held-out data.

Problem

Dataset inference for LLMs requires an in-distribution held-out (validation) set known to be absent from training, which is rarely available; slight distribution shifts inflate false positives.

Approach

Generate the held-out set with a generator trained on suffix completion over the suspect data (short 32-64 token snippets, shared prefix), then calibrate by comparing a text-only real-vs-synthetic classifier against a membership-aware classifier (adding target-model MIA scores), using a difference-comparison t-test so only membership-driven shift counts; also constrains MIA weights to be positive

Key results

• On the Pile, the method yields p<0.05 on all evaluated member subsets and p>0.1 on non-member subsets (e.g., Pile-CC member p=0.002, non-member p=0.99), while original DI without calibration gives p<0.001 on non-members.
• Suffix completion generator reduces GPT2 distribution-shift AUC to 52.2% versus 99.0% for ICL paraphrasing and 58.9% for preference optimization.
• On the single-author blog dataset, combined classifier AUC 55.6% vs text 53.8% for members (p=0.01), and p=0.13 for non-members.

Measured effects reported in the paper

Surprising findings

• Generated held-out text is systematically simpler, lowering perplexity-based MIA scores, so unconstrained linear regression assigns negative weights and inflates false positives; sigmoid weight constraint is needed.
• Preference optimization reduces bag-of-words shift to chance (50.2%) yet a GPT2 classifier still reaches 58.9% AUC.

Limitations

• Calibrated DI yields modest AUC gains (about 1-6 points), and detection depends on sample size; fewer than 1,000 samples gives significance on most but not all datasets.
• Only Pythia models and one fine-tuning setting (one epoch LoRA) on the single-author data were evaluated; training from scratch was not done.
• Only Pile subsets free of copyright issues were evaluated; the generator relies on access to the suspect data itself.

Experimental setup

Datasets: single-author blog posts (1400) The Pile subsets (Pile-CC, Wikipedia, ArXiv, NIH ExPorter, FreeLaw, Ubuntu IRC, PubMed Central, Github, EuroParl, PhilPa

Benchmarks: single-author blog posts The Pile subsets

Metrics: AUC (text and combined classifiers) p-value of difference comparison t-test BoW AUC GPT2 AUC

Baselines: ICL paraphrasing ICL text completion preference optimization (SimPO) original DI without calibration

Builds on: dataset inference LLM dataset inference Min-K%++ SimPO Blind Baselines

Closest papers by meaning

• LLM Dataset Inference: Did you train on my dataset? (NeurIPS 2024 · similarity 0.87)
• The Canary’s Echo: Auditing Privacy Risks of LLM-Generated Synthetic Text (ICML 2025 · similarity 0.87)
• Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective (ICLR 2025 · similarity 0.87)
• Valid Inference with Imperfect Synthetic Data (NeurIPS 2025 · similarity 0.86)
• SynBench: A Benchmark for Differentially Private Text Generation (NeurIPS 2026 · similarity 0.86)
• It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs (NeurIPS 2026 · similarity 0.86)