Research map / Federated learning and privacy / Personalized and heterogeneous FL
Private and Personalized Frequency Estimation in a Federated Setting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a clustering-based federated method for personalized frequency (histogram) estimation in KL divergence, with a joint-DP version, private data-dependent initialization, and a Dirichlet-mixture analysis; it beats FedAvg+FT and IFCA on Reddit, StackOverflow, and Amazon Reviews.
Problem
Personalized next-word distribution estimation across users with heterogeneous token distributions, measured in KL divergence, under user-level (joint) differential privacy and varying user dataset sizes.
Approach
Adapt Lloyd-style clustering to KL divergence, average local Good-Turing estimates to form cluster centers, locally finetune centers with a Dirichlet-posterior update, and make it jointly DP via adaptive clipping, noisy sums, and a data-dependent exponential-mechanism initialization.
Key results
- Before finetuning, the method reduces NLL suboptimality gap by at least 25-45% over baselines in the non-private setting and up to 50% in the private setting.
- Under private size-heterogeneous setting on Amazon Reviews, >50% NLL suboptimality reduction versus IFCA.
- Good-Turing averaging and private initialization (Alg. 5) reduce clustering iterations from 200 to 20 in the private setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed clustering-based method instead of standard and clustering baselines | Personalized frequency estimation · Reddit, StackOverflow, Amazon Reviews | test NLL suboptimality gap: over 26% (non-private, averaged over datasets); over 42% (private, averaged over datasets) | Helped |
| Use the proposed method instead of single-global-model baselines | Size-heterogeneous personalized frequency estimation · Reddit, StackOverflow, Amazon Reviews | test NLL suboptimality gap: > 40% error reduction | Helped |
| Use the proposed method instead of IFCA | Private size-heterogeneous frequency estimation · Amazon Reviews | test NLL suboptimality gap: > 50% reduction | Helped |
| Use the proposed method instead of IFCA | Private size-heterogeneous frequency estimation · Reddit, StackOverflow, Amazon Reviews | error: over 30% | Helped |
| Replace random cluster initialization with data-dependent PrivateInit (ablation) | Proposed clustering method · Reddit, StackOverflow, Amazon Reviews | test NLL: 200 to 20 clustering iterations | Helped |
| Use Good-Turing averages instead of empirical averages for cluster centers (ablation) | Proposed clustering method · Reddit, StackOverflow, Amazon Reviews | test NLL | Helped |
| Use the proposed local finetuning update instead of gradient descent or RTFA (ablation) | FedAvg model · Reddit, StackOverflow, Amazon Reviews | test NLL | Helped |
| Set the number of clusters to K=10 rather than tune it separately for each dataset (ablation) | Proposed clustering method · StackOverflow, Amazon Reviews | test NLL | Hurt |
| Vary the number of clusters K (ablation) | Proposed clustering method · Reddit, StackOverflow, Amazon Reviews | test NLL | Mixed |
Surprising findings
- Clustering-based personalization can outperform the popular FedAvg+FT baseline on real-world text distributions, contrary to common belief.
Limitations
- Only estimates the marginal unigram token distribution per user, not conditional next-token distributions.
- The private initialization sampling step is not implemented in a federated manner and needs a central small dataset.
- Theoretical guarantees rely on a stylized Dirichlet mixture model.
Experimental setup
Datasets: Reddit StackOverflow Amazon Reviews
Benchmarks: Reddit StackOverflow Amazon Reviews
Metrics: test negative log likelihood KL suboptimality gap
Baselines: FedAvg FedAvg+FT MAML IFCA RTFA
Builds on: Lloyd's clustering IFCA FedAvg Good-Turing estimator joint differential privacy
Closest papers by meaning
- PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs (ICML 2024 · similarity 0.85)
- Leveraging Randomness in Model and Data Partitioning for Privacy Amplification (ICML 2025 · similarity 0.85)
- Private Model Personalization Revisited (ICML 2025 · similarity 0.85)
- Beyond Memorization: Violating Privacy via Inference with Large Language Models (ICLR 2024 · similarity 0.85)
- A Generalized Shuffle Framework for Privacy Amplification: Strengthening Privacy Guarantees and Enhancing Utility (AAAI 2024 · similarity 0.84)
- Private Adaptive Covariance Estimation via Gaussian Graphical Models (NeurIPS 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.