Research map / Large language models / Instruction and fine-tuning
Can We Infer Confidential Properties of Training Data from LLMs?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces a benchmark (built on ChatDoctor) for property inference attacks on fine-tuned LLMs under QA (SFT) and chat-completion (CLM-FT) modes, and proposes a black-box prompt-based generation attack and a grey-box shadow-model attack using word-frequency features.
Problem
Dataset-level confidential properties (e.g., patient gender ratio, diagnosis prevalence) in domain-specific LLM fine-tuning data can leak through the fine-tuned model, but it is unclear whether property inference attacks developed for discriminative and generative image/tabular models transfer to LL
Approach
Adapts generation-based property inference to LLMs via designed prompts with labeled outputs, and introduces a shadow-model meta-regressor (XGBoost) over word-frequency features of generated text, selected via f_regression over keywords, to predict property ratios directly.
Key results
- In Q&A mode, word-frequency shadow attack gave lowest gender MAE for Llama-1 (11.43 at 30% female target vs 36.52 for BB generation), e.g., 6.85 at 70% vs 1.45 for BB generation.
- In chat-completion mode, BB generation achieved gender MAE as low as 0.61 (Llama-3, 30% target) and 1.33-3.28 for Llama-1 across targets, outperforming baselines.
- For medical diagnosis, BB generation consistently beat direct asking and generation w/o FT across mental, digestive, and childbirth properties; childbirth had highest error (e.g., 7.73-7.95 MAE in Q&A for Llama models).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use word-frequency shadow features instead of BB generation in Q&A Mode | Gender property inference · Gender, Q&A Mode | MAE | Helped |
| Use BB generation instead of word-frequency attack | Gender property inference · Gender, Chat-Completion Mode | MAE | Helped |
| Use BB generation rather than baseline attacks | Medical diagnosis property inference · Medical diagnosis, Q&A and Chat-Completion Modes | MAE | Helped |
| Infer childbirth rather than mental or digestive disorder | BB generation attack · Medical diagnosis | MAE | Hurt |
| Increase the number of selected keywords from too few toward an intermediate value (ablation) | Word-frequency attack · Gender, Q&A and Chat-Completion Modes | MAE | Mixed |
| Increase the number of shadow models (ablation) | Word-frequency attack · Gender, Q&A and Chat-Completion Modes | MAE | Helped |
| Generate more samples (ablation) | BB generation attack · Gender, Chat-Completion Mode | MAE: with 1000 generated samples, the mean absolute error (MAE) drops below 4% for all three model architectures | Helped |
| Aggregate three prompts rather than rely on a single prompt (ablation) | BB generation attack · Gender, Chat-Completion Mode | MAE | Helped |
| Use Llama-3 rather than the target's Llama-1 as the shadow-model base (ablation) | Word-frequency shadow-model attack · Gender, Q&A Mode | MAE: 6.86% -> 20.23%; 12.96% -> 0.24%; 6.82% -> 20.00% | Mixed |
| Use Llama-3 rather than the target's Llama-1 as the shadow-model base (ablation) | Word-frequency shadow-model attack · Gender, Chat-Completion Mode | MAE: 3.44% -> 3.85%; 0.27% -> 0.17%; 6.26% -> 15.31% | Mixed |
| Adjust generation temperature from T = 1 to T = 0.5, T = 1.5, or T = 3.5 (ablation) | Llama-3 BB-generation defense · Gender, Chat-Completion Mode | Predicted female ratio: 0.355 -> 0.200; 0.516 -> 0.447; 0.6958 -> 0.8686 | Hurt |
Surprising findings
- Pretrained Pythia (and fine-tuned Pythia for direct asking) fails to produce usable output for direct prompts, so direct asking could not be evaluated.
- Childbirth property is hardest to infer, possibly because pretrained models suppress childbirth-related content due to safety training.
Limitations
- Low attack success for the childbirth property.
- Evaluated only on ChatDoctor-based medical dialogue data and three models.
- Temperature-based defense is simple and may fail if the adversary knows the temperature or has model weights.
Experimental setup
Datasets: ChatDoctor Alpaca iCliniq
Benchmarks: PropInfer
Metrics: mean absolute error BERTScore
Baselines: Direct asking Perplexity attack Generation w/o FT
Builds on: ChatDoctor Alpaca LoRA XGBoost
Closest papers by meaning
- Fragments to Facts: Partial-Information Fragment Inference from LLMs (ICML 2025 · similarity 0.89)
- Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.87)
- Private Attribute Inference from Images with Vision-Language Models (NeurIPS 2024 · similarity 0.87)
- LLM Dataset Inference: Did you train on my dataset? (NeurIPS 2024 · similarity 0.87)
- Remarkable Robustness of LLMs: Stages of Inference? (NeurIPS 2025 · similarity 0.87)
- Hidden No More: Attacking and Defending Private Third-Party LLM Inference (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.