Research map / Federated learning and privacy / Differential privacy
GeoClip: Geometry-Aware Clipping for Differentially Private SGD
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GeoClip, a DP-SGD variant that clips and noises gradients in a transformed basis (estimated from previously released noisy gradients at no extra privacy cost), with a closed-form optimal transform and a streaming low-rank PCA variant.
Problem
Setting the per-sample gradient clipping threshold in DP-SGD is hard; existing adaptive clipping methods (AdaClip, quantile-based) operate in the standard coordinate basis and ignore correlations between gradient coordinates, wasting noise.
Approach
Clip and perturb gradients after a centering and linear transform M_t (estimated from past privatized gradients' mean and covariance), with a closed-form optimal M_t that minimizes injected noise subject to a control on clipping probability; generalizes AdaClip from diagonal to full-rank transforms.
Key results
- On Breast Cancer at epsilon 0.67, GeoClip reaches 87.87±3.32% test accuracy vs 84.90±5.91% for AdaClip and 81.41±12.53% for quantile-based clipping.
- On Fashion-MNIST final-layer fine-tuning at epsilon 0.6, GeoClip achieves 73.09±0.72% vs 71.78±1.28% for quantile-based and 68.35±0.41% for AdaClip.
- On the synthetic Gaussian regression task GeoClip converges by about epoch 2, while the next-best method (quantile-based) needs nearly twice as many epochs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GeoClip instead of adaptive clipping baselines | Private training on Diabetes · Diabetes test at ε = 0.50, 0.86, and 0.93 | test MSE: 0.073$\pm$ 0.015 vs 0.077$\pm$0.027; 0.044$\pm$0.003 vs 0.062$\pm$0.028; 0.039$\pm$0.009 vs 0.055$\pm$0.014 | Helped |
| Use GeoClip instead of adaptive clipping baselines | Private training on Breast Cancer · Breast Cancer test at ε = 0.67, 0.8, and 0.87 | test accuracy (%): 87.87$\pm$3.32 vs 84.90$\pm$5.91; 88.57$\pm$3.37 vs 85.42$\pm$5.34; 93.63$\pm$1.63 vs 87.71$\pm$5.87 | Helped |
| Use GeoClip instead of adaptive clipping baselines | Private training on Android Malware · Malware test at ε = 0.26, 0.49, and 0.67 | test accuracy (%): 90.77$\pm$1.83 vs 88.35$\pm$3.27; 91.64$\pm$1.26 vs 90.55$\pm$1.55; 92.67$\pm$1.63 vs 90.57$\pm$1.61 | Helped |
| Use GeoClip instead of clipping baselines | Final-layer DP fine-tuning of an MNIST-pretrained CNN · Fashion-MNIST at ε = 0.6 and ε = 1 | accuracy: 73.09$\pm$0.72 vs 71.78$\pm$1.28; 73.09$\pm$0.63 vs 72.09$\pm$1.12 | Helped |
| Vary γ from 0.2 to 0.6 to 1.0 (ablation) | GeoClip (k = 50) · USPS | test accuracy (%): 85.537; 86.182; 85.752 | No clear effect |
| Vary h₂ from 1 to 10, 100, and 1000 (ablation) | GeoClip (k = 50) · USPS | test accuracy (%): 86.021; 85.752; 85.752; 85.752 | No clear effect |
| Vary h₁ from 10⁻⁶ to 10⁻¹⁰ and 10⁻¹⁵ (ablation) | GeoClip (k = 50) · USPS | test accuracy (%): 85.752; 85.752; 85.752 | No clear effect |
Surprising findings
- Ablations show the scaling hyperparameter gamma and clamp floor h1 have little effect on USPS accuracy, so the method's sensitivity is mostly to h2.
Limitations
- Requires an eigendecomposition per step, adding computation and memory; low-rank variant mitigates this.
- Introduces extra hyperparameters (notably h2) that must be tuned.
- Evaluated only on a limited set of small datasets and small models.
Experimental setup
Datasets: Diabetes Breast Cancer Android Malware USPS MNIST Fashion-MNIST synthetic Gaussian
Benchmarks: Diabetes Breast Cancer Android Malware Fashion-MNIST USPS synthetic Gaussian
Metrics: test MSE test accuracy (%) epsilon spent
Baselines: AdaClip Quantile-based clipping DP-SGD
Builds on: DP-SGD AdaClip quantile-based clipping Connect-the-Dots accountant
Closest papers by meaning
- Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach (ICLR 2024 · similarity 0.89)
- Differentially Private Stochastic Gradient Descent with Fixed-Size Minibatches: Tighter RDP Guarantees with or without Replacement (NeurIPS 2024 · similarity 0.87)
- Online Sensitivity Optimization in Differentially Private Learning (AAAI 2024 · similarity 0.87)
- The importance of feature preprocessing for differentially private linear optimization (ICLR 2024 · similarity 0.86)
- DP-SGD Without Clipping: The Lipschitz Neural Network Way (ICLR 2024 · similarity 0.85)
- Clipped SGD Algorithms for Performative Prediction: Tight Bounds for Stochastic Bias and Remedies (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.