Research map / Large language models / Language, safety and interpretability
LLM Dataset Inference: Did you train on my dataset?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that membership inference attacks on LLMs mostly detect distribution shift rather than membership, and proposes dataset inference that aggregates many MIA scores with a linear regressor and a statistical t-test to detect whether a suspect dataset was used to train Pythia models.
Problem
Membership inference attacks on LLMs trained for a single epoch on web-scale data appear successful only due to distribution shift between members and non-members (e.g., WikiMIA temporal shift), and fail on IID splits; a method is needed to attribute whole datasets instead of individual sequences.
Approach
Instead of classifying individual sequences, aggregate features from many MIAs (52 in total) over a suspect dataset, learn per-distribution weights with a linear regressor on private validation vs. suspect splits, and run a t-test on held-out samples, with p-values combined across random seeds.
Key results
- Dataset inference achieves p-values below 0.1 for distinguishing Pile train vs validation splits across all subsets with 1000 points on Pythia-12B.
- On validation-vs-validation comparisons (false positive check), dataset inference yields no false positives, with p-values above 0.1 in all cases.
- Individual MIAs such as Min-k% Prob get AUC near 0.5 on IID Wikipedia splits of the Pile, while reversing train/val in WikiMIA gives AUC 0.7, showing distribution shift confounds MIA evaluation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Aggregate MIA features selectively with a linear regressor and test the held-out scores | Pythia-12B · Pile train vs. validation subsets | p-value: below 0.1 | Helped |
| Apply dataset inference to two validation subsets rather than train versus validation | Pythia-12B · Pile validation subset vs. validation subset | p-value: higher than 0.1 | No clear effect |
| Evaluate Min-k% Prob on IID train and validation splits instead of temporally shifted WikiMIA data | Pythia · Pile Wikipedia IID train vs. validation | AUC: close to 0.7 -> close to 0.5 | Hurt |
| Use Pile validation examples as the suspect set against post-2023 Wikipedia | Min-k% Prob on Pythia · Pile Wikipedia validation vs. post-2023 Wikipedia | AUC: 0.7 | Hurt |
| Use one MIA across different data distributions | Membership inference attacks · Pile subsets | AUC: less than 0.5 | Mixed |
| Use a single MIA feature instead of selectively combining features with a linear model (ablation) | Dataset inference · Pile subsets | dataset inference success | Mixed |
| Replace outlier removal and normalization with mean correction or outlier clipping (ablation) | Dataset inference · Pile subsets | false positives | Hurt |
| Normalize feature values (ablation) | Dataset inference · Pile subsets | maximum p-value | Helped |
| Increase the number of data points (ablation) | Dataset inference · Pile subsets | p-value | Helped |
| Use a larger language model (ablation) | Dataset inference · Pile subsets | p-value | Mixed |
| Use a non-deduplicated rather than deduplicated training set (ablation) | Dataset inference · Pile subsets with 500 query points | p-value | Helped |
Surprising findings
- Min-k% Prob achieves AUC of about 0.7 when comparing Pile validation data against post-2023 Wikipedia, even though validation data was never trained on, revealing that MIAs detect temporal distribution shift.
- Perplexity-based and other MIAs often show AUC below 0.5 on some Pile subsets, implying false positives labeling non-members as members.
Limitations
- Requires suspect and validation sets to be IID and the validation set to remain private to the victim.
- Requires gray-box access to model loss; not applicable to black-box APIs without log-probs.
- Dataset inference works best for larger models and for non-deduplicated training data; less reliable for smaller models and some subsets.
Experimental setup
Datasets: The Pile (20 subsets) WikiMIA
Benchmarks: The Pile (20 subsets, train vs validation) WikiMIA
Metrics: AUC p-value false positive rate (validation vs validation p-values)
Baselines: Min-k% Prob DetectGPT perplexity thresholding zlib ratio reference model based MIAs perturbation based MIAs
Builds on: dataset inference (Maini et al. 2021) Min-k% Prob DetectGPT zlib ratio
Closest papers by meaning
- Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration (NeurIPS 2024 · similarity 0.91)
- DF-MIA: A Distribution-Free Membership Inference Attack on Fine-Tuned Large Language Models (AAAI 2025 · similarity 0.91)
- MIA-Tuner: Adapting Large Language Models as Pre-training Text Detector (AAAI 2025 · similarity 0.89)
- Membership Inference Attacks against Large Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- Language Models May Verbatim Complete Text They Were Not Explicitly Trained On (ICML 2025 · similarity 0.88)
- Fragments to Facts: Partial-Information Fragment Inference from LLMs (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.