Research map / Learning theory and algorithms / Prediction and decision losses
Provable Training Data Identification for Large Language Models
Zhenlong Liu, Hao Zeng, Weiran Huang, Hongxin Wei
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add p-value scaling with JKBB estimate of member proportion (ablation) | conformal BH · WikiMIA NeoX-20B MIN-K% alpha=0.1 | power (%): 1.69 -> 5.90 | Improved |
| add p-value scaling with JKBB estimate of member proportion (ablation) | conformal BH · WikiMIA NeoX-20B perplexity alpha=0.2 | power (%): 2.37 -> 10.88 | Improved |
| add p-value scaling with JKBB estimate of member proportion (ablation) | conformal BH · WikiMIA NeoX-20B MIN-K% target FIR 0.5 | power: 0.5177 -> 0.8618 | Improved |
| use JKBB estimator instead of Storey estimator (ablation) | PTDI · WikiMIA NeoX-20B MIN-K% alpha=0.3 | power (%): 42.81 -> 53.31 | Improved |
| use JKBB estimator instead of BKY (ablation) | PTDI · WikiMIA LLaMA-7B MIN-K% alpha=0.3 | power (%): 8.47 -> 18.47 | Improved |
| use JKBB estimator instead of Quantile-BH (ablation) | PTDI · WikiMIA Pythia-6.9B MIN-K% alpha=0.3 | power (%): 10.25 -> 24.24 | Improved |
| use PTDI instead of knockoff-based KTD | FIR control · WikiMIA and XSum, white-box knockoff statistic | FIR | Improved |
| use PTDI instead of prior approach | FIR control · WikiMIA Pythia-1.4B target FIR 0.05 | empirical FIR: 0.1311 -> 0.0494 | Improved |
| increase calibration-to-test ratio rho from 0.1 to 1 (ablation) | PTDI · ArXivTection | FIP variance | Improved |
| vary training member proportion pi_test (ablation) | PTDI · VL-MIA/Flickr MiniGPT-4 | FIR | No clear effect |
| add adjusted moment estimator using known members (ablation) | PTDI · WikiMIA | power | Improved |
Closest papers by meaning
- Unlocking Post-hoc Dataset Inference with Synthetic Data (ICML 2025 · similarity 0.86)
- Large language model validity via enhanced conformal prediction methods (NeurIPS 2024 · similarity 0.85)
- Detecting Pretraining Data from Large Language Models (ICLR 2024 · similarity 0.84)
- Training on the Benchmark Is Not All You Need (AAAI 2025 · similarity 0.84)
- Provable Joint Decontamination for Benchmarking Multiple Large Language Models (NeurIPS 2026 · similarity 0.84)
- Beyond Model Collapse: Scaling Up with Synthesized Data Requires Verification (ICLR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
