Research map / Federated learning and privacy / Privacy attacks and secure learning
PANORAMIA: Privacy Auditing of Machine Learning Models without Retraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PANORAMIA, a post-hoc privacy leakage measurement that audits a trained model using known training members versus generated (synthetic) non-members, without retraining or modifying the model or training data.
Problem
Measuring the privacy leakage of a trained ML model with respect to a known subset of its training data, post-hoc, when in-distribution non-member data is unavailable and retraining or control of the training pipeline is impossible.
Approach
Replace real non-members with samples from a generative model trained on a subset of the members, train a loss-based membership inference classifier to separate known members from generated non-members, and compare it against a baseline that lacks access to the target model; the gap yields a privacy measurement (c-closeness plus epsilon, reported as epsilon-tilde) rather than a strict lower bound
Key results
- On CIFAR-10 ResNet101 trained 100 epochs, PANORAMIA measures epsilon-tilde of 1.16 ± 0.19, close to RM;RN (1.22) and O(1) (1.41).
- On GPT-2 trained 150 epochs, PANORAMIA's measurement rises to 2.40 ± 0.53 versus 5.73 for O(1), with the test-set size becoming a bottleneck around 20k samples.
- On DP ResNet18, PANORAMIA detects no leakage at epsilon=1 and 2 and increasing leakage at epsilon=10, 15 and infinity (1.43 at infinity).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase ResNet101 training from 20 to 100 epochs | PANORAMIA audit of CIFAR10 ResNet101 · CIFAR10 | measured privacy loss ε̃: 0.33 ± 0.21 -> 1.16 ± 0.19 | Helped |
| Increase WRN-28-10 training from 150 to 300 epochs | PANORAMIA audit of CIFAR10 WRN-28-10 · CIFAR10 | measured privacy loss ε̃: 2.57 ± 0.23 -> 3.54 ± 0.23 | Helped |
| Increase CNN training from 50 to 100 epochs | PANORAMIA audit of CelebA CNN · CelebA | measured privacy loss ε̃: 0.76 ± 0.23 -> 1.33 ± 0.22 | Helped |
| Increase GPT-2 training from 37 to 150 epochs | PANORAMIA audit of GPT-2 · WikiText | measured privacy loss ε̃: 0.22 ± 0.37 -> 2.40 ± 0.53 | Helped |
| Increase the DP-SGD privacy parameter ε from 1 to 15 | PANORAMIA audit of ResNet18 · CIFAR10 | measured privacy loss ε̃: 0.00 -> 1.13±0.19 | Helped |
| Increase the DP-SGD privacy parameter ε from 15 to ∞ | PANORAMIA audit of ResNet18 · CIFAR10 | measured privacy loss ε̃: 1.13±0.19 -> 1.43±0.21 | Helped |
| Use 20k test samples instead of 10k | PANORAMIA audit of ResNet101 trained for 100 epochs · CIFAR10 | measured privacy loss ε̃: 2.23 -> 2.62 | Helped |
| Increase test set size to 20k | PANORAMIA audit of GPT-2 trained for 150 epochs · WikiText | measured privacy loss ε̃: 2.59 | No clear effect |
| Use a helper model trained on synthetic data rather than no helper (ablation) | PANORAMIA baseline on CIFAR10 · CIFAR10 | c_lb: +1.19 | Helped |
| Train the helper on synthetic rather than real non-member data (ablation) | PANORAMIA baseline on CIFAR10 · CIFAR10 | c_lb: +0.23 | Helped |
| Use larger auditing test sets | PANORAMIA audit of DP ResNet18 · CIFAR10 | measured privacy loss ε̃ | Helped |
| Apply PANORAMIA to tabular classification | PANORAMIA · Tabular data | privacy leakage measurement | No clear effect |
Surprising findings
- On tabular data, PANORAMIA did not detect meaningful leakage and returned high-variance, inconclusive audit values.
- The one-sided c-closeness definition makes the generator-based audit weaker than O(1), which benefits from non-member detection, so the best prior methods measure larger leakage.
Limitations
- The measurement epsilon-tilde is not a lower bound on epsilon-DP; it is only close to one when the baseline is tight and the generator closeness c is small.
- Tabular data results were inconclusive due to high variance.
- Test-set size bounds the power of the hypothesis test; GPT-2 measurements saturate around 20k test samples.
Experimental setup
Datasets: CIFAR-10 CelebA WikiText-103
Benchmarks: CIFAR-10 CelebA WikiText-103
Metrics: epsilon-tilde (privacy measurement) c_lb lower bound precision at recall test accuracy
Baselines: O(1) audit RM;RN (real non-member MIA)
Builds on: O(1) privacy auditing DP-SGD StyleGAN2 GPT-2
Closest papers by meaning
- Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.89)
- Forget to Flourish: Leveraging Machine-Unlearning on Pretrained Language Models for Privacy Leakage (AAAI 2025 · similarity 0.87)
- Auditing Private Prediction (ICML 2024 · similarity 0.86)
- Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models (NeurIPS 2024 · similarity 0.86)
- One-shot Empirical Privacy Estimation for Federated Learning (ICLR 2024 · similarity 0.85)
- Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.