AI Paper Atlas

Research map / Federated learning and privacy / Differential privacy

Sparsity-Preserving Differentially Private Training of Large Embedding Models

Badih Ghazi, Yangsibo Huang, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Amer Sinha, Chiyuan Zhang

NeurIPS 2023 · Federated learning and privacy › Differential privacy · approach: DP-SGD refinements · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use DP-AdaFEST instead of LoRA for embedding gradient sizeRoBERTa word embeddings · SST-2 eps=1.0gradient size reduction: 5.91 -> 14.59 at utility loss 0.001Improved
use DP-AdaFEST instead of LoRA at utility loss 0.01RoBERTa word embeddings · SST-2 eps=1.0gradient size reduction: 47.28 -> 53.90Improved
larger vocabulary model (XLM-R 250k vs RoBERTa 50k)DP-AdaFEST · XNLI vs SST-2 eps=1.0gradient size reduction: 53.90 -> 152.94 at utility loss 0.01Improved
combine DP-FEST preselection with DP-AdaFEST (DP-AdaFEST+)DP-AdaFEST · Criteo-Kagglegradient size reduction: >10^6Improved
combine DP-FEST with DP-AdaFESTDP-AdaFEST · Criteo-time-seriesutility/efficiency trade-offImproved
apply exponential selection sparsity to DP-SGDDP-SGD · Criteo-Kaggle, SST-2, QNLI, QQPutilityHurt
DP-AdaFEST instead of DP-FESTDP-FEST · Criteo-Kagglegradient size reduction at same utilityImproved
DP-AdaFEST instead of DP-FEST on time-series dataDP-FEST · Criteo-time-series eps=1.0gradient reduction at same utilityImproved
DP-FEST frequency from streaming sums instead of first day only (ablation)DP-FEST · Criteo-time-seriesutilityImproved
streaming-based instead of all-day frequency for DP-FEST (ablation)DP-FEST · Criteo-time-seriesutilityNo clear effect
increase noise ratio sigma1/sigma2 (ablation)DP-AdaFEST · Criteo-Kaggle and SST-2accuracy: best utility at ratio 5 or 10Improved
increase noise ratio sigma1/sigma2 (ablation)DP-AdaFEST · Criteo-Kaggle and SST-2gradient densityHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy