Research map / Federated learning and privacy / Differential privacy
DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiSK, a simplified Kalman-filter denoiser for differentially private optimizers that combines gradients at two points via a finite-difference Hessian-vector approximation and exponential averaging, improving DP-SGD/DP-Adam(W) accuracy at the same privacy budget.
Problem
Differentially private optimizers suffer large accuracy drops in large-scale training because the injected Gaussian noise (scaled with model dimension and iterations) disrupts optimization dynamics.
Approach
Model the true gradient as a dynamic state with privatized gradients as noisy observations, then apply a Kalman filter simplified to a scalar constant gain (identity covariance, finite-difference Hessian-vector product), yielding a single exponentially weighted gradient estimate that needs one extra forward pass and two extra optimizer states.
Key results
- Training from scratch under epsilon=8: ImageNet-1k accuracy rises from 33.56% to 36.89% and CIFAR-100 from 21% to 42% (reported 41.8%), exceeding prior SOTA under the same privacy budget.
- Fine-tuning RoBERTa-base on GLUE at epsilon=1: KF-DP improves over DPAdamW on all tasks by at least 3.8% (e.g., MNLI 80.7 to 82.0 for KF-DP, 84.7 with KF-DPLora).
- CIFAR-10 from-scratch accuracy improves from 63% to 75% under the same privacy budget.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DiSK to the DP optimizer | ImageNet-1k training from scratch · ImageNet-1k validation | accuracy: 33.56% -> 36.89% | Helped |
| Apply DiSK to the DP optimizer | CIFAR-100 training from scratch · CIFAR-100 | accuracy: 40.6% -> 41.8% | Helped |
| Apply DiSK to the DP optimizer | CIFAR-10 training from scratch · CIFAR-10 | accuracy: 63% -> 75% | Helped |
| Apply DiSK to the DP optimizer | CIFAR-100 fine-tuning · CIFAR-100 fine-tuning | accuracy: 85% -> 89% | Helped |
| Apply DiSK to DPAdamW | RoBERTa-base · GLUE MNLI, ε=6.7 | accuracy: 83.2 -> 84.8 | Helped |
| Apply DiSK to DPAdamW | RoBERTa-base · GLUE QNLI, ε=6.7 | accuracy: 87.5 -> 88.9 | Helped |
| Apply DiSK to DPAdamW | RoBERTa-base · GLUE SST-2, ε=6.7 | accuracy: 91.5 -> 92.8 | Helped |
| Apply DiSK to DPAdamW | RoBERTa-base · GLUE QQP, ε=6.7 | accuracy: 85.8 -> 88.5 | Helped |
| Apply DiSK to DPAdamW | RoBERTa-base · GLUE MNLI, ε=1 | accuracy: 80.7 -> 82.0 | Helped |
| Apply DiSK to DPAdamW | RoBERTa-base · GLUE QNLI, ε=1 | accuracy: 86.0 -> 88.7 | Helped |
| Apply DiSK to DPAdamW | RoBERTa-base · GLUE SST-2, ε=1 | accuracy: 91.4 -> 91.5 | Helped |
| Apply DiSK to DPAdamW | RoBERTa-base · GLUE QQP, ε=1 | accuracy: 84.2 -> 86.9 | Helped |
Limitations
- Theory assumes bounded per-sample gradients (clipping inactive) and uses a constant-covariance simplification of the Kalman filter.
- Requires one additional forward pass per step, up to twice the compute of DP-SGD and up to 2x memory of DPSGD.
- Hyperparameters such as kappa and gamma require tuning via grid search per privacy budget.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100 ImageNet-1k GLUE E2E DART synthetic dataset
Benchmarks: MNIST CIFAR-10 CIFAR-100 ImageNet-1k GLUE E2E DART
Metrics: test accuracy validation accuracy
Baselines: DPSGD DPAdam DPAdamW DOPPLER DP-LoRA
Builds on: DPSGD DP-Adam Kalman filter NAG STORM
Closest papers by meaning
- DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reduction (NeurIPS 2024 · similarity 0.92)
- Towards hyperparameter-free optimization with differential privacy (ICLR 2025 · similarity 0.90)
- The importance of feature preprocessing for differentially private linear optimization (ICLR 2024 · similarity 0.88)
- Differentially Private Stochastic Gradient Descent with Fixed-Size Minibatches: Tighter RDP Guarantees with or without Replacement (NeurIPS 2024 · similarity 0.88)
- Privacy without Noisy Gradients: Slicing Mechanism for Generative Model Training (NeurIPS 2024 · similarity 0.88)
- Optimal Differentially Private Model Training with Public Data (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.