Research map / Federated learning and privacy / Differential privacy
Auditing Private Prediction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the first framework for empirically auditing private prediction algorithms (PATE, CaPC, PromptPATE, Private-kNN) via RDP auditing under poisoning and query-control adversaries.
Problem
Private prediction (noising predictions at inference rather than training) lacks any empirical auditing technique, so the gap between theoretical DP upper bounds and actual privacy leakage is unknown.
Approach
Instantiate adversaries by crafter type (natural vs poisoning) and distinguisher type (natural vs adversarial queries), audit per-query RDP via the 2-cut of Renyi divergence with Clopper-Pearson intervals, compose losslessly across queries, and compare against an exact Renyi divergence computation for noisy argmax.
Key results
- Exact RDP computation for noisy argmax is tighter than the data-dependent PATE-style analysis, indicating slack in the analysis.
- Adversaries with poisoning and adversarial queries leak the most; PATE shows least leakage, followed by CaPC and PromptPATE, with PromptPATE near-maximal leakage under poisoning.
- Privacy leakage for natural-query adversaries (poisoning crafter, natural distinguisher) is much lower than for adversarial-query adversaries across all algorithms, with negligible leakage for PATE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use exact Renyi divergence instead of the prior data-dependent theoretical calculation | Private prediction privacy analysis · PATE, CaPC, PromptPATE | privacy-analysis tightness | Helped |
| Give the adversary control over test queries instead of restricting it to natural queries | Private prediction · PATE, CaPC, PromptPATE | privacy leakage | Hurt |
| Allow a poisoning crafter instead of using a natural crafter | PATE, CaPC, PromptPATE · Adversarial repeated queries | privacy leakage | Hurt |
| Use poisoning with natural queries instead of poisoning with adversarial repeated queries | PATE, CaPC, PromptPATE · Natural queries | privacy leakage | Helped |
| Use a mislabelled poison instead of a natural in-distribution point | PATE · Adversarial repeated queries | privacy leakage | Hurt |
| Use deterministic data partitioning instead of random partitioning | Private prediction · PATE versus CaPC | privacy leakage | Hurt |
| Use a worst-case adversarial embedding instead of a poison point from the train or test data | Private-kNN · Text classification | privacy leakage | Hurt |
| Use the 2-cut audit instead of the exact Renyi divergence calculation | Noisy argmax · Synthetic neighboring histograms | audit tightness | Hurt |
| Use k-cut or bootstrap audits instead of 2-cut confidence-interval audits | Noisy argmax · Synthetic neighboring histograms | audit tightness | Mixed |
Surprising findings
- Natural-query adversaries leak negligible privacy for PATE, far below the theoretical bound.
- The exact RDP calculation is tighter than the existing data-dependent analysis, showing the published privacy analysis is loose.
Limitations
- Parametric multinomial assumptions replace full retraining of teachers in each audit experiment.
- Poisoning attacks are simple mislabelled points; stronger attacks on natural queries remain open.
- Bootstrap-based audits are unreliable for small Renyi divergences, so the main results rely on 2-cut confidence-interval audits.
Experimental setup
Datasets: MNIST CIFAR10 Fashion MNIST SST2 AGNEWS DBPedia TREC
Benchmarks: MNIST CIFAR10 Fashion MNIST SST2 AGNEWS DBPedia TREC
Metrics: RDP privacy leakage epsilon at delta=1e-6 2-cut audit lower bound exact RDP divergence
Baselines: data-dependent PATE analysis data-independent subsampled Gaussian analysis
Builds on: PATE CaPC PromptPATE Private-kNN 2-cut Renyi divergence auditing
Closest papers by meaning
- Auditing $f$-differential privacy in one run (ICML 2025 · similarity 0.91)
- Nearly Tight Black-Box Auditing of Differentially Private Machine Learning (NeurIPS 2024 · similarity 0.91)
- How Well Can Differential Privacy Be Audited in One Run? (NeurIPS 2025 · similarity 0.90)
- The Limits of Differential Privacy in Online Learning (NeurIPS 2024 · similarity 0.90)
- Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model (ICLR 2025 · similarity 0.89)
- One-shot Empirical Privacy Estimation for Federated Learning (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.