Research map / Federated learning and privacy / Differential privacy

Improved Communication-Privacy Trade-offs in $L_2$ Mean Estimation under Streaming Differential Privacy

ICML 2024 · Federated learning and privacy › Differential privacy · approach: Private training with public data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an L2-geometry Rényi DP accountant for the sparsified (coordinate-subsampled) Gaussian mechanism and extends it to streaming DP via matrix factorization (SGMF) for DP-FTRL, achieving about 100x compression in FL tasks.

Problem

Joint communication and central differential privacy constraints for L2 mean estimation in federated learning, where prior sparsification schemes are optimized for L_infinity geometry and do not extend to streaming DP (tree aggregation / matrix factorization, DP-FTRL).

Approach

Treat the randomness of coordinate sparsification as part of the DP noise and account privacy directly in L2 geometry by exploiting convexity of the Rényi DP profile of the 1-D subsampled Gaussian mechanism, then extend the accountant to adaptive streaming releases by decoupling the transcript so the matrix mechanism becomes non-adaptive.

Key results

• The L2-CSGM accountant's Rényi DP converges to that of the uncompressed Gaussian mechanism as Delta_inf/Delta_2 goes to 0, with the ratio d*Delta_inf^2/Delta_2^2 showing L_inf-CSGM is suboptimal by a factor that is O(log d) under random rotation.
• Empirically, L2-CSGM achieves over 100x compression on F-EMNIST and Stack Overflow next-word prediction for DP-SGD with little accuracy loss, with compression improving as cohort size grows.
• For DP-FTRL, SGMF compression rates are generally lower than for DP-SGD, and performance is more sensitive to server learning rate and L2 clip norm.

Measured effects reported in the paper

Surprising findings

• For DP-FTRL matrix mechanisms, achievable compression is generally lower than for DP-SGD-type DP-FedAvg, and results are more sensitive to hyperparameters.
• Compression impact on utility diminishes as cohort size grows, so greater compression is feasible with larger cohorts.

Limitations

• For matrix mechanisms the achievable compression rate is lower and performance is sensitive to server learning rate and clip norm.
• Analysis assumes cohort size 1 in the main text (general cohort via appendix) and single participation per epoch with mechanism restart.
• Experiments on a smaller CNN variant (d <= 2^20) and rely on random rotation with L_inf clipping; the privacy bound for the streaming case is stated in a form that is tight only under the stated assumptions.

Experimental setup

Datasets: Federated EMNIST Stack Overflow next-word prediction

Benchmarks: Federated EMNIST Stack Overflow next-word prediction

Metrics: accuracy mean squared error Rényi DP / (epsilon, delta)-DP compression rate

Baselines: uncompressed Gaussian mechanism L_infinity-CSGM uncompressed matrix mechanism (MF)

Builds on: CSGM (coordinate subsampled Gaussian mechanism) DP-FTRL matrix mechanism factorization (Denisov et al.) DP-FedAvg Rényi DP

Closest papers by meaning

• Privacy Amplification via Compression: Achieving the Optimal Privacy-Accuracy-Communication Trade-off in Distributed Mean Estimation (NeurIPS 2023 · similarity 0.91)
• Universal Exact Compression of Differentially Private Mechanisms (NeurIPS 2024 · similarity 0.90)
• Keeping a Secret Requires a Good Memory: Unconditional Streaming Lower-Bounds for Differentially Private Algorithms (ICML 2026 · similarity 0.89)
• A Huber Loss Minimization Approach to Mean Estimation under User-level Differential Privacy (NeurIPS 2024 · similarity 0.88)
• 𝑓-Differential Privacy Filters: Validity and Approximate Solutions (NeurIPS 2026 · similarity 0.87)
• Privacy Guarantees in Posterior Sampling under Contamination (NeurIPS 2026 · similarity 0.87)