Research map / Federated learning and privacy / Differential privacy
A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a private adaptive HPO method that tunes a single scalar r = learning rate × steps using cheap low-epsilon trials, fits a line in epsilon, and scales r linearly to the target privacy budget, accounting for HPO privacy cost.
Problem
Hyperparameter optimization for DP-SGD is costly and its privacy cost is usually not accounted for, since standard grid/random search requires many trials that consume privacy budget.
Approach
Combine learning rate and number of iterations into a single radius r = eta x T, estimate optimal r at small epsilons via cheap trials, fit a line r(eps) and extrapolate to the target epsilon, so HPO costs only a small fraction of the privacy budget.
Key results
- On CIFAR10 without public data at epsilon=1, the method reaches 62.63% versus 44% for random search and 68% for grid search (which does not count HPO privacy cost).
- On ImageNet at epsilon=1 (including HPO privacy cost), the method reports 83.2%, with 87% cited in the text for the ViT setting, exceeding prior DP results cited in the paper under the same budget.
- Linear scaling on CIFAR10 with public data yields 70.5% at epsilon=1, above the prior results of 67% and 66% cited in the paper.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use linear-scaling adaptive HPO instead of random search | CIFAR10 without public data · CIFAR10 | accuracy: 44 -> 62.63 | Helped |
| use linear-scaling adaptive HPO instead of random search | CIFAR100 · CIFAR100 | accuracy: 84.44 -> 89.10 | Helped |
| use linear-scaling adaptive HPO instead of random search | ImageNet · ImageNet | accuracy: 81.2 -> 86.7 | Helped |
| use linear-scaling adaptive HPO instead of random search | SQuAD with public data · SQuAD | accuracy: 49.33 -> 78.08 | Helped |
| use linear-scaling adaptive HPO instead of prior private HPO | CIFAR10 linear probing · CIFAR10, ε=1 | accuracy: 67% -> 70.5%; 66% -> 70.5% | Helped |
| use data-dependent linear-scaling HPO instead of DPAdamWOSM | ImageNet · ImageNet, ε=1 | accuracy: 79% -> 87% | Helped |
| use a linear fit with 3 points rather than 2 (ablation) | linear-scaling HPO · ImageNet | mean accuracy: 86.69 (0.86) -> 87.81 (0.86) | Helped |
| use a quadratic fit with 3 points rather than a linear fit with 2 (ablation) | linear-scaling HPO · ImageNet | mean accuracy: 86.69 (0.86) -> 86.64 (1.08) | No clear effect |
| add batch size and clipping threshold to the optimized hyperparameters (ablation) | linear-scaling HPO · CIFAR100 | accuracy: 89.10 -> 87.9 ± 1.9 | Hurt |
| add a classifier without bias (ablation) | CIFAR100 private training · CIFAR100, ε=0.1 | accuracy improvement: 0.36 | Helped |
| use zero initialization instead of random initialization (ablation) | CIFAR100 private training · CIFAR100, ε=0.1 | accuracy improvement: 6.81 | Helped |
| use gradient descent instead of SGD with batch size 4096 (ablation) | CIFAR100 private training · CIFAR100, ε=0.1 | accuracy improvement: 1.46 | Helped |
Surprising findings
- Increasing noise to reduce epsilon is far more effective than subsampling for reducing privacy cost on ImageNet (84.07% vs 41.34% at epsilon=0.7).
- Fine-tuning a DP model improved OOD accuracy on Waterbirds despite degrading in-distribution accuracy.
Limitations
- Runs are serial, so wall-clock time is worse than random search and parallelization is worse than grid search.
- For small epsilon on hard datasets like ImageNet, HPO trials with small budgets provide little information, so the HPO privacy cost can consume most of the budget.
- The theory relies on strong convexity, smoothness, and bounded gradient assumptions that do not hold for neural networks.
Experimental setup
Datasets: ImageNet CIFAR10 CIFAR100 FashionMNIST STL10 EMNIST SQuAD SST-2 QNLI QQP MNLI PersonaChat
Benchmarks: ImageNet CIFAR10 CIFAR100 FashionMNIST STL10 EMNIST SQuAD SST-2 QNLI QQP MNLI PersonaChat
Metrics: accuracy perplexity relative error rate reduction privacy cost (epsilon)
Baselines: random search grid search DPAdamWOSM Rényi DP private HPO methods DP-Adam prior work on ImageNet
Builds on: DP-SGD PLD accounting random search Rényi DP private HPO
Closest papers by meaning
- Towards hyperparameter-free optimization with differential privacy (ICLR 2025 · similarity 0.91)
- DP-HyPO: An Adaptive Private Framework for Hyperparameter Optimization (NeurIPS 2023 · similarity 0.91)
- Online Sensitivity Optimization in Differentially Private Learning (AAAI 2024 · similarity 0.88)
- Private Zeroth-Order Optimization with Public Data (NeurIPS 2025 · similarity 0.87)
- Differentially private optimization for non-decomposable objective functions (ICLR 2025 · similarity 0.87)
- Delving into Differentially Private Transformer (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.