AI Paper Atlas

Research map / Federated learning and privacy / Differential privacy

Towards Hyperparameter Transfer for Differentially Private Optimization

Tianze Wang, Zhiqi Bu, Linjun Zhang

NeurIPS 2026 · Federated learning and privacy › Differential privacy · approach: DP-SGD refinements · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that Maximal Update Parametrization (μP) fails to enable zero-shot hyperparameter transfer under differential privacy, due to distinct spectral properties of DP-induced noise and clipping.

Problem

Hyperparameter tuning and scaling are costly for large models trained with differential privacy, where noise injection and per-sample gradient clipping complicate optimization; μP's zero-shot transfer is known to work for standard training but has not been shown to hold under DP.

Approach

Identifies that μP breaks under DP because of distinct spectral properties introduced by DP noise and clipping, motivating a study of how hyperparameters should be transferred for DP optimization. Abstract-only information; the specific remedy is not stated in the provided text.

Key results

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use DP-muP layer-wise SNR-aligned scaling instead of standard parametrization5-layer MLP, DP-SGD/DP-Adam · CIFAR-10 (training loss across widths)training loss stability across widthsImproved
use muP scaling with SNR condition instead of standard parametrizationViT, DP-SGD and DP-Adam · CIFAR-10 and CIFAR-100learning rate transfer across widthsImproved
apply only the muP spectral condition without the SNR condition under DP noise (ablation)DP-SGD · noise-dominated regimelearning rate transfer across widthsHurt
transfer learning rate from a 76.0M proxy to GPT-2 Large with DP-muPGPT-2 Large · Shakespeare-charcompute cost of LR search: 30 minutes -> 11 minutes per runImproved
use SGD updates for input and output layers instead of Adam updates in Muon (ablation)Muon (non-private and DP) · CIFAR-10 MLPlearning rate transferImproved
scale noise multiplier proportionally to batch size with fixed iteration countDP-muP, MLP and ViT · CIFAR-10 batch size transfertraining loss curves alignmentImproved
use clipping-only muP rules under DP-SGD in the noise-dominated regime (ablation)DP-SGD · theorysignal update scale: signal update scales as Theta(1/sqrt(n))Hurt

Surprising findings

Limitations

Experimental setup

Builds on: Maximal Update Parametrization (μP)

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy