Research map / Federated learning and privacy / Differential privacy

Differentially private optimization for non-decomposable objective functions

ICLR 2025 · Federated learning and privacy › Differential privacy · approach: Private training with public data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Logit-DP, a DP-SGD variant that clips pairwise similarity gradients so that the sensitivity of the summed contrastive (and spreadout) loss gradient is O(1) in batch size, and shows better CIFAR-10/100 results than naive DP-SGD.

Problem

DP-SGD for non-decomposable similarity-based losses such as contrastive loss: the L2 sensitivity of the batch gradient grows with batch size, so the Gaussian noise needed for privacy scales badly with n.

Approach

Decompose the batch gradient of the contrastive/spreadout loss into per-pair similarity gradients, clip each pairwise similarity gradient to norm B, and add Gaussian noise to the aggregated sum; under cosine similarity and the stated conditions the sensitivity is bounded by a constant times B independent of n.

Key results

• On CIFAR-10 pre-training with the embedding net, Logit-DP reaches 0.819 relative accuracy versus 0.827 for Naive-DP (both relative to non-private), while on ResNet18 Logit-DP reaches 0.730 relative accuracy versus 0.599 for Naive-DP.
• On CIFAR-100 fine-tuning with frozen embeddings, Logit-DP achieves relative accuracy 1.013 versus 0.946 for Naive-DP, with both at epsilon=5.
• Naive-DP's training loss remains nearly constant during pre-training for the embedding net, whereas Logit-DP reduces the loss.

Measured effects reported in the paper

Surprising findings

• Naive-DP's relative accuracy on CIFAR-100 fine-tuning slightly exceeds its recall-matched performance on the embedding net while having much lower precision, and Logit-DP's relative accuracy exceeds the non-private baseline (1.013).

Limitations

• Gradient materialization of n^2 pairwise similarity gradients creates a memory and runtime bottleneck; efficient norm computation is left as future work.
• Experiments use small models (generic embedding net, ResNet18 without BatchNorm) and CIFAR-scale datasets, with single-GPU training and only a single run for CIFAR-100.
• Sensitivity bounds rely on cosine similarity and specific loss forms (contrastive, spreadout); the guarantees do not directly cover other similarity functions.

Experimental setup

Datasets: CIFAR-10 CIFAR-100

Benchmarks: CIFAR-10 CIFAR-100

Metrics: accuracy recall precision F-beta score relative training loss k-NN accuracy (k=3)

Baselines: Naive-DP Non-private SGD

Builds on: DP-SGD contrastive loss spreadout regularizer Gaussian mechanism

Closest papers by meaning

• Differentially Private Prototypes for Imbalanced Transfer Learning (AAAI 2025 · similarity 0.88)
• Private Zeroth-Order Optimization with Public Data (NeurIPS 2025 · similarity 0.88)
• The importance of feature preprocessing for differentially private linear optimization (ICLR 2024 · similarity 0.88)
• Differentially Private Stochastic Gradient Descent with Fixed-Size Minibatches: Tighter RDP Guarantees with or without Replacement (NeurIPS 2024 · similarity 0.88)
• Certification for Differentially Private Prediction in Gradient-Based Training (ICML 2025 · similarity 0.87)
• DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reduction (ICLR 2025 · similarity 0.87)