Research map / Federated learning and privacy / Differential privacy

Enabling Differentially Private Federated Learning for Speech Recognition: Benchmarks, Adaptive Optimizers, and Gradient Clipping

NeurIPS 2025 · Federated learning and privacy › Differential privacy · approach: Differentially private federated learning · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Benchmarks federated learning with user-level differential privacy for a 250M-parameter transformer CTC ASR model, and shows per-layer clipping combined with the LAMB server optimizer makes DP training viable, with a convergence bound.

Problem

Federated learning with user-level differential privacy has not been studied for large transformer-based end-to-end ASR, where layer-wise gradient heterogeneity and DP noise hurt convergence.

Approach

Revive per-layer clipping (uniform or dimension-based budget split) and pair it with the layer-wise adaptive LAMB server optimizer, so DP noise no longer dominates attention layers; supported by a convergence bound that couples trust ratio, per-layer clipping, and DP noise.

Key results

• With per-layer clipping, FL with DP at sigma=3e-6 yields 21.2% test WER vs 19.1% without DP, under (7.2, 1e-9)-DP extrapolated to ~70M clients.
• At sigma=1e-5, per-layer clipping gives 23.7% dev WER with (4.5, 1e-9)-DP extrapolated to ~7M clients.
• Global clipping under DP noise degrades WER substantially (e.g., 30.7 dev WER at sigma=1e-5 vs 21.3 with per-layer clipping).

Measured effects reported in the paper

Surprising findings

• Clipping client deltas to C=1e-8 has no impact on model quality because the LAMB trust ratio re-normalizes gradients.
• A LS-960 seed model outperforms a CV-en-train-10 in-domain seed model on the remaining CV data.

Limitations

• Experiments limited to read speech (LibriSpeech and Common Voice), monolingual CTC models of 100M-500M parameters, no external language model.
• Privacy guarantees for large populations are extrapolated rather than trained at full scale; cohort sizes simulated are ~1k.
• Theoretical analysis assumes beta1=0, lambda=0, and q=1 (no sampling) in the main proof.

Experimental setup

Datasets: LibriSpeech Common Voice v13.0

Benchmarks: LibriSpeech dev/test Common Voice English dev/test Common Voice French/German

Metrics: word error rate epsilon at delta=1e-9

Baselines: global clipping FL with DP central training FL without DP Adam server optimizer

Builds on: LAMB DP-FedAvg moments accountant SCAFFOLD FedProx

Closest papers by meaning

• DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models (CVPR 2026 · similarity 0.90)
• Efficient DP-SGD for LLMs with Randomized Clipping (ICML 2026 · similarity 0.88)
• Noise-Aware Algorithm for Heterogeneous Differentially Private Federated Learning (ICML 2024 · similarity 0.87)
• DR-Encoder: Encode Low-rank Gradients with Random Prior for Large Language Models Differentially Privately (AAAI 2025 · similarity 0.87)
• Differentially Private Stochastic Gradient Descent with Fixed-Size Minibatches: Tighter RDP Guarantees with or without Replacement (NeurIPS 2024 · similarity 0.86)
• Thinking Forward: Memory-Efficient Federated Finetuning of Language Models (NeurIPS 2024 · similarity 0.86)