Research map / Federated learning and privacy / Differential privacy
Continual Release Moment Estimation with Differential Privacy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Joint Moment Estimation (JME), which privately releases first and second moments of a data stream via the matrix mechanism with a joint sensitivity analysis so the second moment comes at no extra privacy cost, and applies it to private Gaussian density estimation and DP-Adam.
Problem
Differentially private continual release of first and second moments of a vector data stream, where estimating both naively splits the privacy budget and increases noise.
Approach
Jointly analyze the sensitivity of privatizing the data and its outer product with a scaling parameter lambda chosen so the joint sensitivity equals that of the first moment alone, giving second-moment privacy for free within the matrix mechanism.
Key results
- JME's first-moment error matches prefix-sum baselines while adding second-moment estimation at no extra privacy cost (second-moment privacy for free).
- In the high-privacy regime (sigma=2), private Gaussian density estimation with JME achieves lower KL divergence than PP after about 10 samples.
- On CIFAR-10 at epsilon ~0.16 over 10 epochs, DP-Adam-JME reaches 30.38% accuracy versus 10.48% for vanilla DP-Adam; at epsilon ~1.7 it reaches 47.94% vs 43.07%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use JME instead of vanilla DP-Adam | DP-Adam · CIFAR-10, ε ≈ 0.16, Epoch 10 | accuracy: 10.48 -> 30.38 | Helped |
| Use JME instead of debiased postprocessing | DP-Adam · CIFAR-10, ε ≈ 0.16, Epoch 10 | accuracy: 29.14 -> 30.38 | Helped |
| Use JME instead of joint clipping | DP-Adam · CIFAR-10, ε ≈ 0.16, Epoch 10 | accuracy: 28.14 -> 30.38 | Helped |
| Use JME instead of vanilla DP-Adam | DP-Adam · CIFAR-10, ε ≈ 1.7, Epoch 10 | accuracy: 43.07 -> 47.94 | Helped |
| Use JME instead of joint clipping | DP-Adam · CIFAR-10, ε ≈ 1.7, Epoch 10 | accuracy: 44.34 -> 47.94 | Helped |
| Replace debiased postprocessing with JME | DP-Adam · CIFAR-10, ε ≈ 1.7, Epoch 10 | accuracy: 48.53 -> 47.94 | Hurt |
Surprising findings
- Vanilla DP-Adam essentially fails at epsilon ~0.16 (around 10% accuracy), while JME-based DP-Adam still trains.
Limitations
- JME's advantage is mainly in high-privacy regimes; in low privacy, post-processing with debiasing can be better.
- Sensitivity closed forms assume decreasing column norm noise-shaping matrices.
- Adam experiments are small-scale (CIFAR-10 with a small CNN, 10 epochs).
Experimental setup
Datasets: CIFAR-10 synthetic Wishart-distributed Gaussian data
Benchmarks: CIFAR-10
Metrics: expected approximation error (Frobenius norm) KL divergence test accuracy
Baselines: Independent Moment Estimation (IME) Concatenate-and-split (CS) Post-processing (PP) DP-Adam DP-AdamBC Joint Clipping
Builds on: matrix mechanism Adam DP-Adam Gaussian mechanism
Closest papers by meaning
- A Private Approximation of the 2nd-Moment Matrix of Any Subsamplable Input (NeurIPS 2025 · similarity 0.87)
- Private Adaptive Covariance Estimation via Gaussian Graphical Models (NeurIPS 2026 · similarity 0.86)
- Improved Communication-Privacy Trade-offs in $L_2$ Mean Estimation under Streaming Differential Privacy (ICML 2024 · similarity 0.86)
- Tractable MCMC for Private Learning with Pure and Gaussian Differential Privacy (ICLR 2024 · similarity 0.84)
- Private Statistical Estimation via Truncation (NeurIPS 2025 · similarity 0.84)
- One-shot Empirical Privacy Estimation for Federated Learning (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.