Research map / Federated learning and privacy / Differential privacy
Contrastive Private Data Synthesis via Weighted Multi-PLM Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes WASP, a differentially private synthetic text data framework that fuses multiple PLMs with DP Top-Q voting, contrastive in-context learning, and dynamic PLM importance weighting, for data-scarce settings.
Problem
Generating differentially private synthetic text data with pre-trained language models is hard when only a few hundred private samples are available, since noisy distribution estimates, low-quality synthetic samples, and per-PLM bias degrade the synthetic data.
Approach
Extends Private Evolution's Top-1 voting to a DP Top-Q voting with decaying weights, uses both nearest (high-quality) and furthest (low-quality) voted samples in a contrastive in-context prompt, and weights multiple PLMs by how close their outputs vote to private samples.
Key results
- With 100 private samples and (4.0, 1e-5)-DP, WASP reaches 89.52 on IMDb, 61.21 on Yelp-Rating and 37.10 on Openreview-Rating, outperforming the best single-PLM Aug-PE (Flan-T5: 89.00, 58.69, 35.42).
- In the federated setting (L=10, 300 samples), WASP achieves 89.65 on IMDb and 61.46 on Yelp-Rating, above Pre-Text baselines.
- Ablations show removing contrastive prompting costs 1.56 points on Yelp-Rating, and removing PLM importance weighting costs 2.27 points on Yelp-Rating.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WASP instead of FuseGen | DP synthetic data generation with 6 open-source PLMs, L=1 · IMDb | accuracy: 89.07 -> 89.52 | Helped |
| Use WASP instead of the strongest Aug-PE PLM | DP synthetic data generation with 6 open-source PLMs, L=1 · Yelp-Rating | accuracy: 58.69 -> 61.21 | Helped |
| Use WASP instead of the strongest Pre-Text PLM | federated DP synthetic data generation, L=10 · Openreview-Category | accuracy: 38.31 -> 40.47 | Helped |
| Add low-quality examples to the contrastive prompt (ablation) | WASP · IMDb | accuracy: 89.21 -> 89.52; 0.31% decrease without contrastive prompting | Helped |
| Add low-quality examples to the contrastive prompt (ablation) | WASP · Yelp-Rating | accuracy: 59.65 -> 61.21; 1.56% decrease without contrastive prompting | Helped |
| Add low-quality examples to the contrastive prompt (ablation) | WASP · Openreview-Rating | accuracy: 36.18 -> 37.10; 0.92% decrease without contrastive prompting | Helped |
| Use dynamic PLM importance weights instead of equal weights (ablation) | WASP · IMDb | accuracy: 89.17 -> 89.52; 0.35% decrease without importance weighting | Helped |
| Use dynamic PLM importance weights instead of equal weights (ablation) | WASP · Yelp-Rating | accuracy: 58.94 -> 61.21; 2.27% decrease without importance weighting | Helped |
| Use dynamic PLM importance weights instead of equal weights (ablation) | WASP · Openreview-Rating | accuracy: 35.53 -> 37.10; 1.57% decline without importance weighting | Helped |
| Increase Top-Q votes per private sample from Q=1 to Q=8 (ablation) | WASP · IMDb | accuracy: 89.02 -> 89.52 | Helped |
| Increase Top-Q votes per private sample from Q=1 to Q=8 (ablation) | WASP · Yelp-Rating | accuracy: 58.74 -> 61.21 | Helped |
| Increase the number of collaborating PLMs (ablation) | WASP · Yelp-Rating | accuracy | Helped |
Surprising findings
- Aug-PE with a single PLM often fails under limited private samples (100), e.g., OPT on IMDb gives 83.86 and GPT-2 on Openreview-Rating gives 24.12, showing PLM choice matters a lot.
Limitations
- Evaluated only on classification tasks; the authors note non-classification tasks remain unverified.
- Selection and weighting are sample-level and the authors suggest more precise weighting could help.
- Privacy-utility evaluation uses a single DP setting for most experiments; the stated code link is anonymous.
Experimental setup
Datasets: IMDb Yelp-Category Yelp-Rating Openreview-Category Openreview-Rating Banking77
Benchmarks: IMDb Yelp-Category Yelp-Rating Openreview-Category Openreview-Rating Banking77
Metrics: STM classification accuracy FID
Baselines: OnlyPrivate FuseGen Aug-PE Pre-Text
Builds on: Private Evolution Aug-PE FuseGen DP-SGD sentence-t5
Closest papers by meaning
- PCEvolve: Private Contrastive Evolution for Synthetic Dataset Generation via Few-Shot Private Data and Generative APIs (ICML 2025 · similarity 0.87)
- A Privacy-Preserving Framework for Generative Model-driven Synthetic Datasets (AAAI 2025 · similarity 0.87)
- Synthesizing Privacy-Preserving Text Data via Finetuning *without* Finetuning Billion-Scale LLMs (ICML 2025 · similarity 0.87)
- Private Evolution Converges (NeurIPS 2025 · similarity 0.87)
- Privacy Amplification Persists under Unlimited Synthetic Data Release (NeurIPS 2026 · similarity 0.86)
- Privacy without Noisy Gradients: Slicing Mechanism for Generative Model Training (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.