Research map / Federated learning and privacy / Differential privacy
Towards Hyperparameter Transfer for Differentially Private Optimization
Tianze Wang, Zhiqi Bu, Linjun Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that Maximal Update Parametrization (μP) fails to enable zero-shot hyperparameter transfer under differential privacy, due to distinct spectral properties of DP-induced noise and clipping.
Problem
Hyperparameter tuning and scaling are costly for large models trained with differential privacy, where noise injection and per-sample gradient clipping complicate optimization; μP's zero-shot transfer is known to work for standard training but has not been shown to hold under DP.
Approach
Identifies that μP breaks under DP because of distinct spectral properties introduced by DP noise and clipping, motivating a study of how hyperparameters should be transferred for DP optimization. Abstract-only information; the specific remedy is not stated in the provided text.
Key results
- Abstract claims μP fails to provide zero-shot hyperparameter transfer under DP constraints.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DP-muP layer-wise SNR-aligned scaling instead of standard parametrization | 5-layer MLP, DP-SGD/DP-Adam · CIFAR-10 (training loss across widths) | training loss stability across widths | Improved |
| use muP scaling with SNR condition instead of standard parametrization | ViT, DP-SGD and DP-Adam · CIFAR-10 and CIFAR-100 | learning rate transfer across widths | Improved |
| apply only the muP spectral condition without the SNR condition under DP noise (ablation) | DP-SGD · noise-dominated regime | learning rate transfer across widths | Hurt |
| transfer learning rate from a 76.0M proxy to GPT-2 Large with DP-muP | GPT-2 Large · Shakespeare-char | compute cost of LR search: 30 minutes -> 11 minutes per run | Improved |
| use SGD updates for input and output layers instead of Adam updates in Muon (ablation) | Muon (non-private and DP) · CIFAR-10 MLP | learning rate transfer | Improved |
| scale noise multiplier proportionally to batch size with fixed iteration count | DP-muP, MLP and ViT · CIFAR-10 batch size transfer | training loss curves alignment | Improved |
| use clipping-only muP rules under DP-SGD in the noise-dominated regime (ablation) | DP-SGD · theory | signal update scale: signal update scales as Theta(1/sqrt(n)) | Hurt |
Surprising findings
- μP, which transfers hyperparameters zero-shot in standard training, fails under DP constraints.
Limitations
- Only the abstract was available; experimental details are unknown.
Experimental setup
Builds on: Maximal Update Parametrization (μP)
Closest papers by meaning
- Towards hyperparameter-free optimization with differential privacy (ICLR 2025 · similarity 0.91)
- A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization (ICML 2024 · similarity 0.89)
- On the Provable Separation of Scales in Maximal Update Parameterization (ICML 2025 · similarity 0.88)
- Online Sensitivity Optimization in Differentially Private Learning (AAAI 2024 · similarity 0.88)
- Differentially Private Prototypes for Imbalanced Transfer Learning (AAAI 2025 · similarity 0.88)
- DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reduction (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
