Research map / Federated learning and privacy / Differential privacy
The importance of feature preprocessing for differentially private linear optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows theoretically that DPSGD without feature preprocessing incurs error proportional to the max feature norm for linear classification, and proposes DPSGD-F (private mean-centering, quantile-based bias augmentation, then DPSGD) whose error scales with feature diameter, validated on MNIST, Fashion-MNIST and pretrained CIFAR-100 linear probes.
Problem
Whether private feature preprocessing is necessary for differentially private optimization of linear classifiers, since DPSGD's privacy error scales with the maximum feature norm rather than the dataset diameter.
Approach
Combine private mean estimation (error scaling with diameter), private quantile-based feature translation and augmentation with a bias coordinate, feature clipping, and DPSGD on the preprocessed GLM, so the leading privacy error term depends on diam(D) instead of max norm R.
Key results
- On MNIST at epsilon=1, DPSGD-F reaches 92.0% accuracy versus 87.4% for DPSGD.
- On Fashion-MNIST at epsilon=1, DPSGD-F reaches 84.0% versus 77.2% for DPSGD.
- On CIFAR-100 with ImageNet-pretrained features at epsilon=1, DPSGD-F reaches 71.6% versus 70.6% for DPSGD (73.8% vs 70.6% in the table's second column layout).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add private feature preprocessing before DPSGD | DPSGD · MNIST, ε = 1 | accuracy: 87.4 -> 92.0 | Helped |
| Add private feature preprocessing before DPSGD | DPSGD · MNIST, ε = 2 | accuracy: 89.6 -> 92.3 | Helped |
| Add private feature preprocessing before DPSGD | DPSGD · Fashion-MNIST, ε = 1 | accuracy: 77.2 -> 84.0 | Helped |
| Add private feature preprocessing before DPSGD | DPSGD · Fashion-MNIST, ε = 2 | accuracy: 78.7 -> 84.5 | Helped |
| Add private feature preprocessing before DPSGD | DPSGD with ImageNet-pretrained WRN features · CIFAR-100 (pretrained), ε = 1 | accuracy: 70.6 -> 71.6 | Helped |
| Add private feature preprocessing before DPSGD | DPSGD with ImageNet-pretrained WRN features · CIFAR-100 (pretrained), ε = 2 | accuracy: 73.8 -> 74.4 | Helped |
Surprising findings
- Gains of DPSGD-F appear correlated with the average distance to the mean rather than with diam(D), a point the authors note is not fully explained.
- The best DPSGD-F results use very small privacy budget for preprocessing (epsilon_F of 0.02 or 0.05), suggesting preprocessing uses little budget.
Limitations
- Theory is restricted to linear/GLM classifiers with margin losses; extension to deep models is left open.
- Experiments use a modified algorithm (Gaussian mean shift, no quantile step or feature clipping) that differs from the theoretically analyzed DPSGD-F.
- Hyperparameters are tuned by grid search that does not account for privacy loss in tuning.
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR-100 (ImageNet-pretrained WRN features) ImageNet
Benchmarks: MNIST Fashion-MNIST CIFAR-100 (pretrained features)
Metrics: test accuracy optimality gap (theory)
Baselines: DPSGD non-private SGD
Builds on: DPSGD private mean estimation (instance-optimal) PLD accountant Bassily et al. DP ERM lower bounds
Closest papers by meaning
- Differentially Private Stochastic Gradient Descent with Fixed-Size Minibatches: Tighter RDP Guarantees with or without Replacement (NeurIPS 2024 · similarity 0.89)
- DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reduction (NeurIPS 2024 · similarity 0.89)
- Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach (ICLR 2024 · similarity 0.89)
- Optimal Differentially Private Model Training with Public Data (ICML 2024 · similarity 0.89)
- DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reduction (ICLR 2025 · similarity 0.88)
- Differentially Private Prototypes for Imbalanced Transfer Learning (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.