Research map / Large language models / Language, safety and interpretability
Blackbox Model Provenance via Palimpsestic Membership Inference
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes palimpsestic-memorization-based independence tests that let a model developer (Alice) detect whether a blackbox derivative model or observed text (Bob's) comes from her training run, using the ordering of shuffled training data.
Problem
Determining whether a blackbox derivative language model or a piece of generated text was produced from an open-weight model, given only the developer's training transcript, with exact false-positive control.
Approach
Language models memorize later-seen training examples more strongly (palimpsestic memorization), so correlating Bob's model likelihoods (query setting) or text likelihoods under retrained/partitioned models (observational setting) with training order yields exact permutation-based or t-distribution p-values under shuffled training.
Key results
- Query-setting test on Pythia derivatives gives p-values of at most 1e-8 for all but four 1.4B finetunes (with at least one epoch), and OLMo finetunes all below 1e-4.
- Observational shuffle-based test distinguishes Bob's text from as few as 320 tokens with p-values below 1e-3 on TinyStories models when retraining on the last 2% of data.
- Observational partition-based n-gram test on pythia-6.9b-deduped text needs several hundred thousand tokens (median p-value 2.6e-4 at 19.2M tokens for the end checkpoint).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a reference model to control for natural variation in text likelihood (ablation) | Palimpsestic query test · Pythia-deduped provenance, 20M query tokens | p-value: $2.1\times10^{-1}$ -> $8.0\times10^{-20}$ | Helped |
| Use a reference model from the same family and similar capability (ablation) | Palimpsestic query test · Pythia-deduped provenance, 20M query tokens | p-value: $8.0\times10^{-20}$ | Helped |
| Increase query-token budget | Palimpsestic query test · Model provenance | p-value | Helped |
| Use sequences from Epoch 1 rather than Epoch 0 (ablation) | OLMo-7B query test · OLMo-7B training-order provenance | p-value | Helped |
| Sample sequences from the start of each training example (ablation) | OLMo-7B query test · OLMo-7B training-order provenance | p-value | Helped |
| Increase sequence length while holding the total token budget fixed (ablation) | OLMo-7B query test · OLMo-7B training-order provenance | p-value | No clear effect |
| Use the partitioned-transcript text test rather than querying the model | Palimpsestic provenance testing · Pythia text provenance | p-value: $3.0\times10^{-1}$ at 640K tokens; $2.6\times10^{-4}$ at 19.2M tokens | Mixed |
| Use shuffled-transcript retraining instead of partitioning the transcript | Observational provenance test · TinyStories text provenance | p-value: less than $10^{-3}$ from as few as $320$ tokens | Helped |
| Fine-tune retrained test models on Bob's text before evaluating its likelihood (ablation) | Shuffled-transcript observational test · TinyStories text provenance under Bob fine-tuning | p-value | Helped |
| Increase the amount of data used to retrain the test models (ablation) | Shuffled-transcript observational test · TinyStories text provenance under Bob fine-tuning | p-value | Helped |
| Estimate next-token probabilities from repeated generations rather than using actual probabilities (ablation) | OLMo-7B query test · OLMo-7B training-order provenance | p-value: $1.6\times 10^{-4}/5.6\times 10^{-7}/6.4\times10^{-6}$ for N=1/4/16 | Mixed |
Surprising findings
- pythia-2.8b-deduped on HuggingFace appears to have been trained on the non-deduped Pile (p-value about 1e-60), contrary to its documentation.
- Independently trained models on similar data (e.g., OLMo-7B vs Llama) show no significant correlation, while the heuristic output-similarity baselines can flag independent models as related.
Limitations
- Observational partition-based test requires several hundred thousand tokens and is infeasible for short snippets.
- Observational shuffle-based test requires retraining many models, which is costly at scale and gives approximate rather than exact p-values.
- Requires Alice to have access to her ordered training transcript and to shuffled training data; model developers may be reluctant to disclose data.
Experimental setup
Datasets: The Pile Pythia deduped Pile Dolma v1.5 Dolma v1.7 OLMo-Mix TinyStories
Benchmarks: Pythia finetunes (29 HuggingFace models) OLMo finetunes (12 HuggingFace models) TinyStories
Metrics: p-value Spearman correlation z-score
Baselines: dataset inference canary insertion model fingerprinting output-similarity model comparison
Builds on: dataset inference infini-gram Spearman rank correlation
Closest papers by meaning
- Independence Tests for Language Models (ICML 2025 · similarity 0.88)
- Scalable Influence and Fact Tracing for Large Language Model Pretraining (ICLR 2025 · similarity 0.87)
- Copyright Traps for Large Language Models (ICML 2024 · similarity 0.87)
- Learning the Signature of Memorization in Autoregressive Language Models (NeurIPS 2026 · similarity 0.87)
- Statistical Hypothesis Testing for Auditing Robustness in Language Models (ICML 2025 · similarity 0.87)
- PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.