Research map / Federated learning and privacy / Differential privacy

Differentially Private Model Merging

NeurIPS 2026 · Federated learning and privacy › Differential privacy · approach: Private training with public data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes post-processing random selection (RS) and linear combination (LC) of existing DP models trained on the same data to obtain models meeting any target DP budget without retraining, with RDP and PLD accounting.

Problem

Privacy requirements at deployment change, but re-training DP models for each new target is costly; given several DP models with different privacy/utility tradeoffs on the same dataset, how can one produce models satisfying a new target (ε', δ') without additional training or touching raw data?

Approach

Two data-independent merging mechanisms: RS outputs one input model sampled with probabilities π; LC outputs a weighted average λ-combination. Privacy accounting is derived via RDP (log-sum-exp/Hölder bound for RS, joint-release bound and DP-SGD-specific Gaussian bound for LC) and PLD (convexity of hockey-stick divergence, convolution for LC surrogate), with a mean-estimation case study showing LC

Key results

• On synthetic mean estimation, LC consistently achieves lower MSE than RS at matched privacy under both RDP and PLD accounting; PLD gives tighter epsilon than RDP.
• On MNIST logistic regression and CIFAR-10 ResNet18, RS and LC trace continuous accuracy/privacy Pareto frontiers from three merged DP-SGD models, with PLD epsilon consistently below RDP epsilon.
• In the CIFAR-10 pretraining setting, the merged model sometimes outperforms all individual input models.

Surprising findings

• Merged models can outperform all of their individual input models in the pretraining-based CIFAR-10 setting.
• Without knowledge of the training algorithms, LC's privacy cannot improve over releasing all input models jointly, so LC is not guaranteed to beat RS.

Limitations

• LC accounting is developed only for full-batch DP-SGD; subsampled DP-SGD is left to future work.
• Analysis of nonconvex models is not data-dependent; LC tuning relies on public or non-sensitive validation data.
• Privacy of LC over correlated (recycled) checkpoints is not guaranteed beyond the least private checkpoint.

Experimental setup

Datasets: MNIST CIFAR-10 synthetic Gaussian mean estimation data

Benchmarks: MNIST CIFAR-10 synthetic mean estimation

Metrics: accuracy mean squared error epsilon (privacy budget)

Baselines: joint release composition advanced composition

Builds on: DP-SGD Rényi differential privacy privacy loss distributions model soups stochastic weight averaging

Closest papers by meaning

• Optimal Differentially Private Model Training with Public Data (ICML 2024 · similarity 0.90)
• Privacy Profiles for Private Selection (ICML 2024 · similarity 0.88)
• Provable Privacy with Non-Private Pre-Processing (ICML 2024 · similarity 0.88)
• Differentially Private Sum-Product Networks (ICML 2024 · similarity 0.88)
• Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining (ICML 2024 · similarity 0.88)
• Auditing Private Prediction (ICML 2024 · similarity 0.88)