Research map / Federated learning and privacy / Differential privacy
Multi-Class Support Vector Machine with Differential Privacy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PMSVM, differentially private all-in-one (Crammer-Singer-style) multi-class SVMs via weight perturbation and gradient perturbation, which access each sample once to avoid the privacy-budget cost of OvR/OvO.
Problem
Applying differential privacy to multi-class SVMs is costly because OvR and OvO repeatedly query each sample across binary classifiers, consuming the privacy budget in proportion to the number of classes.
Approach
Use all-in-one multi-class SVMs (CS-SVM-style joint optimization) so each training sample is accessed once; add Gaussian noise to the primal weights (PMSVM-WP) or perform noisy DP-SGD on a smoothed hinge loss with clipping and optional Adam-style updates (PMSVM-GP/AGP), with derived sensitivity bounds.
Key results
- Under weight perturbation at epsilon=1, PMSVM-WP reaches 0.599 on Cornell and 0.884 on USPS versus 0.197 (PrivateSVM) and 0.244 (OPERA) on Cornell and 0.184 on USPS for PrivateSVM.
- Under gradient perturbation, PMSVM-AGP achieves 0.929 on HHAR at epsilon=1 and 0.501 on ISOLET at epsilon=1, outperforming GRPUA (0.851 and 0.060).
- Per-iteration gradient time averages 9.27 ms for PMSVM-GP versus 43.34 ms for GRPUA; on the Vehicle dataset weight perturbation underperforms (0.281 vs 0.331 for OPERA at epsilon=1).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use all-in-one PMSVM-WP instead of OvR weight-perturbation baselines | Multi-class SVM · UCI datasets, ε=1 | accuracy | Mixed |
| Use PMSVM gradient perturbation instead of GRPUA | Multi-class SVM · UCI datasets | accuracy | Helped |
| Add adaptive gradient perturbation (AGP) to standard PMSVM-GP | PMSVM-GP · UCI datasets | accuracy | Mixed |
| Add learning-rate decay to PMSVM-GP (ablation) | PMSVM-GP · UCI datasets | accuracy | Mixed |
| Add learning-rate decay to PMSVM-AGP (ablation) | PMSVM-AGP · UCI datasets | accuracy | Mixed |
| Set clipping threshold R=10 rather than R=1 (ablation) | PMSVM-AGP · Cornell, ε=1 | accuracy: 0.693\,$\pm$\,0.032 -> 0.347\,$\pm$\,0.110 | Hurt |
| Use batch size 32 rather than 128 (ablation) | PMSVM-AGP · Cornell, ε=1 | accuracy: 0.693\,$\pm$\,0.032 -> 0.478\,$\pm$\,0.112 | Hurt |
| Use subsampled batches rather than full-batch gradients (ablation) | PMSVM gradient perturbation · UCI datasets | accuracy | Helped |
Surprising findings
- A plain linear layer with cross-entropy and DP-SGD beats the proposed PMSVM-GP on several datasets (e.g., Cornell and ISOLET at epsilon=4 and 8), and it is the best on some settings.
- Learning-rate decay shows no clear advantage for either strategy across datasets.
Limitations
- Weight-perturbation Vehicle results underperform OPERA, attributed to uniform C across methods and weak all-in-one baseline.
- Experiments are limited to six UCI datasets and mostly OvR-based baselines; authors note further validation is needed in healthcare, face recognition and IoT.
- Sensitivity analysis for weight perturbation uses min-max scaling and focuses on CS-SVM; the bound is a generalized extension with stated tightening only for binary case.
Experimental setup
Datasets: Cornell Dermatology HHAR ISOLET USPS Vehicle
Benchmarks: Cornell Dermatology HHAR ISOLET USPS Vehicle
Metrics: accuracy accuracy gap vs non-private computation time
Baselines: PrivateSVM OPERA GRPUA Linear layer with cross-entropy
Builds on: Crammer-Singer multi-class SVM DP-SGD / moments accountant Analytic Gaussian mechanism Opacus
Closest papers by meaning
- Differential Privacy in Scalable General Kernel Learning via $K$-means Nystr{\"o}m Random Features (NeurIPS 2024 · similarity 0.88)
- Differential Privacy Under Class Imbalance: Methods and Empirical Insights (ICML 2025 · similarity 0.87)
- Optimal Differentially Private Model Training with Public Data (ICML 2024 · similarity 0.87)
- Enhancing Learning with Label Differential Privacy by Vector Approximation (ICLR 2025 · similarity 0.86)
- Noise-Aware Differentially Private Regression via Meta-Learning (NeurIPS 2024 · similarity 0.86)
- Differentially Private Prototypes for Imbalanced Transfer Learning (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.