Research map / Adversarial robustness and security / Attacks on text and graphs
Fragments to Facts: Partial-Information Fragment Inference from LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces partial-information fragment inference (PIFI), a threat model where attackers with a few unordered public fragments infer private fragments from fine-tuned LLMs, and proposes data-blind LR-Attack and PRISM attacks competitive with a data-aware LightGBM classifier.
Problem
Fine-tuned LLMs may leak sensitive fragments of a training sample when an adversary knows only a few unordered public fragments of that individual's record, a weaker assumption than the full or ordered prefix assumed by prior memorization and membership inference attacks.
Approach
Use a likelihood ratio between a target model and a shadow model (LR-Attack), and regularize it with a world-model prior via a Bayesian-style update (PRISM), so that attacks require no labeled in-distribution data.
Key results
- LR-Attack achieves 9.5% TPR at 2% FPR on Qwen-2-7B, and PRISM achieves 11.5% TPR at 5% FPR on Llama-3-8B, competitive with the data-aware Classifier (6.3% at 2% FPR, 14.9% at 5% FPR on the main table).
- For rare fragments (occurring once), LR-Attack reaches 17.5% TPR at 2% FPR versus 10.8% for Classifier, while PRISM is best on common fragments with AUC 0.73.
- DP fine-tuning (epsilon=3) suppresses LR-Attack to 0.9% TPR at 2% FPR while Classifier and PRISM retain about 4% TPR at 2% FPR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune for ten or more epochs rather than one epoch (ablation) | Llama-3-8B under PIFI attacks · Medical summarization | attack success rate | Helped |
| Use LoRA instead of full fine-tuning (ablation) | Llama-3-8B under PIFI attacks · Medical summarization | TPR @ FPR | Hurt |
| Increase model size from 3B to 8B (ablation) | Llama models under PIFI attacks · Medical summarization | ROC-AUC | Helped |
| Replace 75% of accurate prompt fragments with inaccurate fragments (ablation) | Llama-3-8B, one-epoch fine-tune · Medical summarization | ROC-AUC, PRISM: .68 to .65 | Hurt |
| Replace 75% of accurate prompt fragments with inaccurate fragments (ablation) | Llama-3-8B, one-epoch fine-tune · Medical summarization | ROC-AUC, Classifier: .66 to .61 | Hurt |
| Replace 75% of accurate prompt fragments with inaccurate fragments (ablation) | Llama-3-8B, one-epoch fine-tune · Medical summarization | ROC-AUC, LR-Attack: .54 unchanged | No clear effect |
| Use sample-level DP fine-tuning with epsilon=3 (ablation) | Llama-3.2-3B under LR-Attack · Medical summarization | TPR @ 2% FPR: TPR@2% FPR=0.9% | Hurt |
| Use DeepSeek-V3 and R1 probabilities as the world model (ablation) | PIFI attacks on Llama-3-8B · Medical summarization | attack performance | Helped |
| Use LR-Attack rather than PRISM for rare fragments | PIFI attacks on convergence Llama-3-8B · Rare medical fragments | ROC-AUC: 0.57 to 0.77 | Helped |
| Use PRISM rather than LR-Attack for common fragments | PIFI attacks on convergence Llama-3-8B · Common medical fragments | ROC-AUC: 0.61 to 0.73 | Helped |
| Use a legal rather than medical summarization task | PIFI attacks on Qwen-2-7B · Legal summarization | ROC-AUC, PRISM: 0.55 | Hurt |
| Fine-tune Llama-3-8B for ten epochs rather than one epoch (ablation) | Llama-3-8B memorization attack · Medical summarization, 10-word prefix | Recall: 0.255 to 0.423 | Helped |
Surprising findings
- The simple likelihood-ratio LR-Attack is competitive with, and on rare fragments outperforms, a data-aware LightGBM classifier.
- Legal-domain attacks are substantially weaker, with AUC around 0.55-0.61.
Limitations
- DP fine-tuning defense is not extensively evaluated.
- Legal setting attacks are much weaker, and PRISM's Bayesian prior is a heuristic with many approximations.
- Evaluated on only medical and legal summarization datasets with fragments extracted by NER and topic modeling.
Experimental setup
Datasets: MTS-Dialog Free Law Project New York CAP legal case data
Benchmarks: MTS-Dialog Free Law CAP legal
Metrics: TPR @ 2% FPR TPR @ 5% FPR ROC-AUC ROUGE-L BERTScore F1 MMLU
Baselines: Classifier (LightGBM) LR-Attack
Builds on: membership inference attacks (likelihood ratio) extractable memorization Neyman-Pearson lemma LoRA
Closest papers by meaning
- Can We Infer Confidential Properties of Training Data from LLMs? (NeurIPS 2025 · similarity 0.89)
- LLM Dataset Inference: Did you train on my dataset? (NeurIPS 2024 · similarity 0.88)
- Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration (NeurIPS 2024 · similarity 0.87)
- Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space (NeurIPS 2024 · similarity 0.86)
- Protecting Your LLMs with Information Bottleneck (NeurIPS 2024 · similarity 0.86)
- Safeguarding LLMs via Model-Agnostic Latent Safety Signals from Dark Knowledge (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.