Research map / Federated learning and privacy / Differential privacy
Noise-Aware Differentially Private Regression via Meta-Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Meta-trains a Convolutional Conditional Neural Process with an improved Gaussian-DP functional mechanism inside its encoder so it produces differentially private, calibrated predictions in one forward pass.
Problem
Standard DP mechanisms significantly degrade predictive performance in small-data regression, and existing DP fine-tuning approaches need large public datasets or costly test-time optimisation.
Approach
Embed a functional DP mechanism (noise from a Gaussian process added to the SetConv encoder's density and signal channels, with clipping) inside a ConvCNP and meta-train it on simulated data so the decoder learns to undo the DP noise; also tightens the functional mechanism's privacy analysis using Gaussian DP, giving about 25-30% lower noise.
Key results
- DPConvCNP outperforms the tuned DP-SVGP baseline on non-Gaussian sawtooth data and is competitive on Gaussian EQ GP data, while being orders of magnitude faster at test time.
- The GDP-based functional mechanism needs about 25% lower noise standard deviation (paper summary cites roughly 30% lower noise magnitude) than prior bounds for the same privacy guarantee.
- On the !Kung sim-to-real task the DPConvCNP performs much better than DP-SVGP for small N and is better calibrated at epsilon=1.00, delta=1e-3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the GDP-based functional mechanism instead of the classical functional mechanism | Functional mechanism · Functional mechanism privacy analysis | noise standard deviation: at least 25% lower standard deviation | Helped |
| Use the GDP-based functional mechanism instead of the classical functional mechanism | Functional mechanism · Functional mechanism privacy analysis | noise magnitude: ≈ 30% lower noise magnitude | Helped |
| Meta-train with clipping and DP noise in the encoder | DPConvCNP · DP prediction on private context data | prediction calibration | Helped |
| Use DPConvCNP instead of DP-SVGP | DPConvCNP · Non-Gaussian synthetic sawtooth tasks | negative log-likelihood | Helped |
| Use DPConvCNP instead of DP-SVGP | DPConvCNP · Sim-to-real !Kung tasks, small datasets | negative log-likelihood | Helped |
| Use DPConvCNP instead of DP-SVGP | DPConvCNP · Gaussian synthetic tasks | negative log-likelihood | Mixed |
| Use DPConvCNP instead of DP-SVGP | DPConvCNP · DP meta-test inference | runtime: several orders of magnitude faster | Helped |
| Amortise one DPConvCNP across different lengthscales (ablation) | DPConvCNP · Gaussian synthetic tasks | negative log-likelihood | Hurt |
| Amortise one DPConvCNP across privacy budgets (ablation) | DPConvCNP · Synthetic tasks across privacy budgets | performance | Hurt |
| Amortise one DPConvCNP across signal periods (ablation) | DPConvCNP · Non-Gaussian synthetic sawtooth tasks | negative log-likelihood | Hurt |
Surprising findings
- The DP-SVGP's performance can decrease as N increases because it underestimates observation noise, yielding over-smoothed and over-confident predictions.
Limitations
- DPConvCNP does not model dependencies between target outputs.
- Sim-to-real performance depends on quality of simulated data matching real data.
- Simulator diversity trades off against predictive certainty.
Experimental setup
Datasets: synthetic GP data (EQ and Matern-3/2 kernels) synthetic sawtooth waveforms Dobe !Kung dataset
Benchmarks: synthetic EQ GP regression synthetic sawtooth regression Dobe !Kung sim-to-real height/weight prediction
Metrics: negative log-likelihood runtime
Baselines: DP-SVGP non-DP Bayes oracle posterior classical functional mechanism bound RDP-based functional mechanism bound
Builds on: ConvCNP functional mechanism Gaussian differential privacy DP-SGD SVGP
Closest papers by meaning
- Towards hyperparameter-free optimization with differential privacy (ICLR 2025 · similarity 0.90)
- Optimal Differentially Private Model Training with Public Data (ICML 2024 · similarity 0.89)
- Attack-Aware Noise Calibration for Differential Privacy (NeurIPS 2024 · similarity 0.89)
- Revisiting Differentially Private ReLU Regression (NeurIPS 2024 · similarity 0.88)
- Differentially Private Prototypes for Imbalanced Transfer Learning (AAAI 2025 · similarity 0.87)
- DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reduction (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.