Research map / Federated learning and privacy / Differential privacy
Differentially Private Model Compression
Fatemehsadat Mireshghallah, Arturs Backurs, Huseyin A Inan, Lukas Wutschitz, Janardhan Kulkarni
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use unstructured DPIMP to produce a 50%-sparse model | BERT-base · GLUE Avg | average score: 81.9 -> 83.5 | Improved |
| Use unstructured rather than structured DPIMP | BERT-base · GLUE Avg | average score: 81.0 -> 83.5 | Improved |
| Initialize half-BERT from pretrained BERT weights rather than randomly (ablation) | Half-BERT · GLUE Avg | average score: 64.5 -> 80.0 | Improved |
| Add DPKD to randomly initialized half-BERT fine-tuning (ablation) | Half-BERT · GLUE Avg | average score: 64.5 -> 62.9 | Hurt |
| Add DPKD after zero-shot PT initialization (ablation) | Half-BERT · GLUE Avg | average score: 80.0 -> 80.3 | No clear effect |
| Add DPKD after zero-shot FT initialization (ablation) | Half-BERT · GLUE Avg | average score: 79.7 -> 80.0 | No clear effect |
| Add DPKD to pretrained DistilBERT fine-tuning (ablation) | DistilBERT · GLUE Avg | average score: 81.9 -> 81.5 | Hurt |
| Replace BERT-base teacher with BERT-large (ablation) | Half-BERT with zero-shot PT initialization and DPKD · GLUE Avg | average score: 80.3 -> 79.5 | Hurt |
| Use a non-private fine-tuned teacher instead of a private teacher (ablation) | Half-BERT with zero-shot PT initialization and DPKD · GLUE Avg | average score: 80.3 -> 81.1 | Improved |
| Use zero-shot PT initialization rather than random initialization with DPKD (ablation) | Half-BERT · GLUE Avg | average score: 62.9 -> 80.3 | Improved |
Closest papers by meaning
- DR-Encoder: Encode Low-rank Gradients with Random Prior for Large Language Models Differentially Privately (AAAI 2025 · similarity 0.89)
- Split-and-Denoise: Protect large language model inference with local differential privacy (ICML 2024 · similarity 0.88)
- DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer (ICLR 2024 · similarity 0.87)
- Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.87)
- Scaling Laws for Differentially Private Language Models (ICML 2025 · similarity 0.87)
- Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
