Research map / Federated learning and privacy / Differential privacy
Towards hyperparameter-free optimization with differential privacy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyFreeDP, a DP optimizer that automatically sets the learning rate via a privatized GeN (loss-curvature) rule plus automatic clipping, removing learning-rate tuning with under 1% extra privacy budget.
Problem
Learning-rate tuning in differentially private deep learning requires multiple non-private runs that leak data through hyperparameter selection and consume privacy budget; existing learning-rate-free methods perform poorly under DP noise.
Approach
Adapt the GeN learning-rate rule to DP by privatizing both the gradient and the per-sample loss values (with an auto-regressive loss clipping threshold R_l near the loss), fit a quadratic along the descent direction, and combine with automatic per-sample gradient normalization; end-to-end privacy accounting splits the budget between gradient and loss noise.
Key results
- On ViT-Small full fine-tuning at epsilon=1, HyFreeDP reaches 96.36% on CIFAR10 and 91.15% on SVHN, while D-Adaptation and Prodigy collapse (e.g. 23.74% and 27.54% on CIFAR10).
- On GPT-2 E2E at epsilon=3, HyFreeDP matches the grid-searched NonDP-GS baseline (BLEU 0.585 vs 0.583) without tuning.
- Training overhead is under 2x versus a single DP run even at update interval K=1, and about 1.1x at K=10 in the reported efficiency table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HyFreeDP instead of DP-hyper | ViT-Small full fine-tuning · SVHN, ε=1 | accuracy: 34.58 -> 91.15 | Helped |
| Use HyFreeDP instead of DP-hyper | ViT-Small full fine-tuning · GTSRB, ε=1 | accuracy: 28.23 -> 74.68 | Helped |
| Use HyFreeDP instead of NonDP-GS | ViT-Small full fine-tuning · Food101, ε=3 | accuracy: 74.58 -> 75.24 | Helped |
| Use HyFreeDP instead of NonDP-GS | GPT-2 full fine-tuning · E2E, ε=3 | BLEU: 0.583 -> 0.585 | Helped |
| Use HyFreeDP instead of NonDP-GS | GPT-2 full fine-tuning · E2E, ε=3 | CIDEr: 1.566 -> 1.564 | No clear effect |
| Use HyFreeDP instead of NonDP-GS | GPT-2 full fine-tuning · E2E, ε=8 | BLEU: 0.612 -> 0.612 | No clear effect |
| Update the learning rate every iteration rather than less frequently (ablation) | HyFreeDP · CIFAR100, ViT-Base | training time (minutes): 152.617 ($\times$ 1.370) -> 118.483 ($\times$ 1.063) -> 113.317 ($\times$ 1.017) | Hurt |
| Update the learning rate every iteration rather than less frequently (ablation) | HyFreeDP · E2E, GPT2 | training time (minutes): 163.333 ($\times$ 1.888) -> 97.167 ($\times$ 1.123) -> 94.983 ($\times$ 1.098) | Hurt |
| Use a constant linear learning-rate scheduler with NonDP-GS (ablation) | ViT full fine-tuning · CIFAR10, ε=1, ViT-Small | accuracy: 96.49 -> 88.02 | Hurt |
| Use a constant linear learning-rate scheduler with NonDP-GS (ablation) | ViT full fine-tuning · CIFAR100, ε=1, ViT-Small | accuracy: 79.70 -> 3.46 | Hurt |
Surprising findings
- Off-the-shelf non-DP learning-rate-free methods (D-Adaptation, Prodigy) perform very poorly under DP, often near chance on CIFAR100 and GTSRB.
- Simply adding a linear learning-rate scheduler to NonDP-GS substantially hurts DP full fine-tuning, showing DP training is sensitive to the schedule.
Limitations
- Loss privatization consumes privacy budget, so the method needs about 1% more gradient noise to match the vanilla DP-SGD budget.
- Results rely on a narrow hand-picked search range for DP-hyper baseline and a 10x learning-rate scaling for LoRA, so hyperparameter-free claims involve some manual scaling choices.
- Evaluated only on ViT, GPT-2 and LLaMA2-7B with a single PubMed subset; broader generality is not established.
Experimental setup
Datasets: CIFAR10 CIFAR100 SVHN GTSRB Food101 E2E PubMed
Benchmarks: CIFAR10 CIFAR100 SVHN GTSRB Food101 E2E PubMed
Metrics: test accuracy BLEU CIDEr METEOR NIST ROUGE_L training time
Baselines: NonDP-GS D-Adaptation Prodigy DP-hyper NonDP-GS w/ LS
Builds on: GeN automatic clipping DP-SGD D-adaptation Prodigy
Closest papers by meaning
- A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization (ICML 2024 · similarity 0.91)
- DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reduction (NeurIPS 2024 · similarity 0.91)
- DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reduction (ICLR 2025 · similarity 0.90)
- Online Sensitivity Optimization in Differentially Private Learning (AAAI 2024 · similarity 0.90)
- Delving into Differentially Private Transformer (ICML 2024 · similarity 0.90)
- Noise-Aware Differentially Private Regression via Meta-Learning (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.