Research map / Federated learning and privacy / Privacy attacks and secure learning
FedRW: Efficient Privacy-Preserving Data Reweighting for Enhancing Federated Learning of Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FedRW, a privacy-preserving federated soft-deduplication method that reweights samples by secure global frequency (via pairwise 2PC PSI with parallel orchestration) instead of deleting duplicates, without a trusted third party.
Problem
Duplicated samples in federated LLM training hurt generalization and privacy, but existing federated deduplication (EP-MPD) hard-deletes samples and relies on a trusted third party for encryption and counting.
Approach
Replace hard deduplication with soft deduplication: compute each sample's global frequency across clients via pairwise two-party PSI-based frequency exchange (PPMPR), organized in a parallel hierarchical schedule, and down-weight frequent samples with a log-based weight in a sample-weighted loss, then aggregate with FedAvg.
Key results
- PPMPR achieves 17.61-28.78x speedup over EP-MPD(I) in preprocessing with 10 to 50 clients (30% duplication, 2^19 samples per client).
- With GPT-2 Large at 30% duplication, FedRW reduces perplexity on Haiku from 2.89 (baseline) to 2.56, a relative reduction of about 11.42%.
- FedRW lowers perplexity under Non-IID label skew from 2.44 to 1.66 on Qwen3-0.6B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace hard deduplication with frequency-aware sample reweighting | GPT-2 Large · Haiku test, 30% duplication | perplexity: 2.89 -> 2.56 | Helped |
| Replace hard deduplication with frequency-aware sample reweighting | GPT-2 Large · Rotten Tomatoes test, 30% duplication | perplexity: 2.21 -> 1.61 | Helped |
| Replace hard deduplication with frequency-aware sample reweighting | GPT-2 Large · Short Jokes test, 30% duplication | perplexity: 3.79 -> 3.15 | Helped |
| Replace hard deduplication with frequency-aware sample reweighting | GPT-2 Large · Sonnets test, 30% duplication | perplexity: 4.35 -> 4.07 | Helped |
| Replace hard deduplication with frequency-aware sample reweighting | DistilGPT2 · Short Jokes test, 30% duplication | perplexity: 2.07 -> 2.08 | Hurt |
| Replace hard deduplication with frequency-aware sample reweighting | DistilGPT2 · Plays test, 30% duplication | perplexity: 16.09 -> 14.50 | Helped |
| Replace hard deduplication with frequency-aware sample reweighting | Qwen3-0.6B · Quantity Skew | perplexity: 2.02 -> 1.96 | Helped |
| Replace hard deduplication with frequency-aware sample reweighting | Qwen3-0.6B · Label Skew | perplexity: 2.44 -> 1.66 | Helped |
| Replace hard deduplication with frequency-aware sample reweighting | Qwen3-0.6B · Feature Skew | perplexity: 3.43 -> 2.70 | Helped |
| Use FedRW PPMPR and parallel orchestration instead of EP-MPD | Federated preprocessing, 10–50 clients · Protocol preprocessing | speedup: 17.61× to 28.78× | Helped |
| Increase duplication from 10% to 90% | Π₂PC, 2^19 data per client · Protocol preprocessing | running time: 620.1_{±12.213} -> 665.9_{±18.913} | No clear effect |
| Increase learning rate across tested values | FedRW · Plays test set | perplexity | No clear effect |
Surprising findings
- Under FedRW, perplexity on Haiku and Rotten Tomatoes stays flat or slightly improves as duplication rises from 10% to 30%, suggesting controlled redundancy can help generalization.
- On DistilGPT2 with Short Jokes, FedRW does not improve over the raw data baseline (2.08 vs 2.07).
Limitations
- Security proofs are only in the semi-honest model without collusion; malicious adversaries are out of scope.
- Experiments use small public text datasets with synthetically injected duplicates across 10 clients; baseline numbers for runtime and perplexity are taken from the original EP-MPD paper rather than rerun.
- Semantic (near-duplicate) redundancy is not addressed; only exact-match frequencies are used.
Experimental setup
Datasets: Haiku Rotten Tomatoes Short Jokes Poetry IMDB Sonnets Plays Twitter Sentiment Analysis
Benchmarks: Haiku Rotten Tomatoes Short Jokes Poetry IMDB Sonnets Plays Twitter Sentiment Analysis
Metrics: perplexity protocol running time (ms) speedup over EP-MPD
Baselines: EP-MPD Raw Data (no deduplication)
Builds on: FedAvg EP-MPD DupLESS RHO-1 DoReMi
Closest papers by meaning
- Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix (NeurIPS 2025 · similarity 0.88)
- FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models (NeurIPS 2025 · similarity 0.87)
- FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language Models (ICML 2024 · similarity 0.87)
- Splitting with Importance-aware Updating for Heterogeneous Federated Learning with Large Language Models (ICML 2025 · similarity 0.87)
- Little Is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learning (AAAI 2025 · similarity 0.86)
- NoT: Federated Unlearning via Weight Negation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.