Research map / Federated learning and privacy / Privacy attacks and secure learning
Rethinking DP-SGD in Discrete Domain: Exploring Logistic Distribution in the Realm of signSGD
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DP-SIGNLOSGD, which replaces the Gaussian noise in DP-signSGD with additive Logistic noise derived from an exponential mechanism over sign sampling, and shows tighter privacy accounting and higher MNIST accuracy.
Problem
Differentially private sign-based SGD (DP-SIGNSGD) typically uses Gaussian noise designed for DP-SGD, which is poorly matched to sign-based updates, leading to loose privacy loss and reduced utility under the same privacy budget.
Approach
Derive the sign-sampling exponential mechanism over {-1,1}^N and show it is equivalent to adding i.i.d. Logistic noise before the sign function, which gives a tighter moments-accountant bound and a lower sign error rate than Gaussian noise.
Key results
- On MNIST, DP-SIGNLOSGD outperforms DP-SIGNSGD across models; e.g. VIT-B-16 test accuracy 99.01% vs 85.46% and VGG11 98.95% vs 97.66% at (epsilon, delta)=(6.4, 1e-5).
- Under the same noise std of 3.0, the Logistic mechanism allows 22 and 71 more training epochs than Gaussian at epsilon 2.0 and 4.0, respectively.
- Logistic noise yields a lower sign error rate than Gaussian and Laplace noise for input gradient magnitudes below about 14 sigma (about 9 percentage points lower error at epsilon 4.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Gaussian noise with additive Logistic noise before the sign function | DP-SIGNSGD with Dense · MNIST test | accuracy: 92.55% -> 95.78% | Helped |
| Replace Gaussian noise with additive Logistic noise before the sign function | DP-SIGNSGD with ResNet-10 · MNIST test | accuracy: 96.48% -> 98.49% | Helped |
| Replace Gaussian noise with additive Logistic noise before the sign function | DP-SIGNSGD with ResNet-18 · MNIST test | accuracy: 96.490% -> 98.70% | Helped |
| Replace Gaussian noise with additive Logistic noise before the sign function | DP-SIGNSGD with ResNet-34 · MNIST test | accuracy: 96.83% -> 98.55% | Helped |
| Replace Gaussian noise with additive Logistic noise before the sign function | DP-SIGNSGD with ResNet-50 · MNIST test | accuracy: 94.92% -> 98.17% | Helped |
| Replace Gaussian noise with additive Logistic noise before the sign function | DP-SIGNSGD with VGG11 · MNIST test | accuracy: 97.66% -> 98.95% | Helped |
| Replace Gaussian noise with additive Logistic noise before the sign function | DP-SIGNSGD with VGG13 · MNIST test | accuracy: 97.98% -> 99.03% | Helped |
| Replace Gaussian noise with additive Logistic noise before the sign function | DP-SIGNSGD with VGG16 · MNIST test | accuracy: 97.63% -> 98.80% | Helped |
| Replace Gaussian noise with additive Logistic noise before the sign function | DP-SIGNSGD with VGG19 · MNIST test | accuracy: 97.58% -> 98.80% | Helped |
| Replace Gaussian noise with additive Logistic noise before the sign function | DP-SIGNSGD with ViT-B-16 · MNIST test | accuracy: 85.46% -> 99.01% | Helped |
| Use additive Logistic noise instead of Gaussian noise | DP-SIGNSGD · sign error rate | sign error rate | Helped |
| Use additive Logistic noise instead of Gaussian noise | DP-SIGNSGD · privacy accountant, noise standard deviation 3.0 and ϵ=2.0 | additional trainable epochs: 22 more training epochs | Helped |
Surprising findings
- The Laplace mechanism with a standard deviation of 70.7 is much noisier than the others, so the authors exclude it from the composition comparison because it is loose under composition.
Limitations
- Privacy analysis for the Logistic mechanism relies on manual numerical integration of the MGF; the closed-form bound is not used in practice.
- Experiments use only MNIST and CIFAR-10 with a single trainable parameter in the privacy analysis, and the convergence and accuracy comparisons are limited to these settings.
- The comparison is primarily against DP-SIGNSGD with Gaussian noise; the paper does not evaluate against stronger modern DP-SGD baselines.
Experimental setup
Datasets: MNIST CIFAR-10
Benchmarks: MNIST CIFAR-10
Metrics: classification accuracy sign error rate privacy loss (epsilon, delta) via moments accountant trainable epochs
Baselines: DP-SIGNSGD (Gaussian noise) Laplace mechanism Gaussian mechanism
Builds on: DP-SGD DP-SIGNSGD signSGD moments accountant exponential mechanism
Closest papers by meaning
- Noisy SIGNSGD Is More Differentially Private Than You (Might) Think (ICML 2025 · similarity 0.90)
- Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model (ICLR 2025 · similarity 0.87)
- Nearly Tight Black-Box Auditing of Differentially Private Machine Learning (NeurIPS 2024 · similarity 0.87)
- Convex Approximation of Two-Layer ReLU Networks for Hidden State Differential Privacy (NeurIPS 2025 · similarity 0.87)
- Proactive DP: A Multiple Target Optimization Framework for DP-SGD (ICML 2024 · similarity 0.86)
- Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.