Research map / Adversarial robustness and security / Attacks on text and graphs
Can Private Machine Learning Be Fair?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that robust federated learning aggregators (Krum, trimmed mean, weak DP) can introduce unfairness, and introduces an update prediction attack that injects unfairness into federated training.
Problem
Whether private (federated, DP) model training can be simultaneously fair and robust to malicious clients, given that robust aggregation may eliminate uncommon-but-benign client models and attackers may target fairness.
Approach
A new "update prediction" attack that lets a malicious client set its update so the aggregated global model equals an attacker-chosen target model that is accurate only on a target subset, without requiring convergence; plus the argument that anomaly-based robust aggregators inherently overlap with benign minority-data models and thus reduce fairness.
Key results
- The update prediction attack raises fairness disparity (∆fairness) to 53.44 on Census, 85.24 on CIFAR-10 and 118.94 on Reddit with no defence.
- Krum and trimmed mean block the attack on CIFAR-10 (∆fairness 6.26 and 6.28) but weak-DP does not (85.29).
- On a constructed dataset, Krum and trimmed mean both reject the benign group B client, reducing its accuracy from 100% to 50%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply the update prediction fairness attack | FedAvg without defence · UCI Census | ∆fairness: 0.00 -> 53.44 | Hurt |
| Apply the update prediction fairness attack | FedAvg without defence · CIFAR-10 | ∆fairness: 0.00 -> 85.24 | Hurt |
| Apply the update prediction fairness attack | FedAvg without defence · Reddit | ∆fairness: 0.00 -> 118.94 | Hurt |
| Use Krum to defend against the fairness attack | FedAvg · CIFAR-10 | ∆fairness: 85.24 -> 6.26 | Helped |
| Use trimmed mean to defend against the fairness attack | FedAvg · CIFAR-10 | ∆fairness: 85.24 -> 6.28 | Helped |
| Use weak-DP to defend against the fairness attack | FedAvg · CIFAR-10 | ∆fairness: 85.24 -> 85.29 | No clear effect |
| Use Krum to defend against the fairness attack | FedAvg · UCI Census | ∆fairness: 53.44 -> -1.72 | Helped |
| Use trimmed mean to defend against the fairness attack | FedAvg · UCI Census | ∆fairness: 53.44 -> 1.82 | Helped |
| Use weak-DP to defend against the fairness attack | FedAvg · UCI Census | ∆fairness: 53.44 -> 2.23 | Helped |
| Use weak-DP to defend against the fairness attack | FedAvg · Reddit | ∆fairness: 118.94 -> 108.39 | Helped |
| Use robust aggregation instead of no defence | Federated training on the synthetic OR dataset · Synthetic OR dataset, Group B | accuracy (%): 100 -> 50 | Hurt |
| Use unfair-update detection instead of no defence | Federated training on the synthetic XOR dataset · Synthetic XOR dataset, Group D | accuracy (%): 100 -> 75 | Hurt |
Surprising findings
- Weak differential privacy, used as a robustness defence, failed to stop the fairness attack under any tested hyperparameter configuration.
- Robust aggregators (Krum, trimmed mean) can increase accuracy disparity even without any attack, because benign minority-data clients look like outliers.
Limitations
- Theoretical convergence proof only covers strongly convex functions; non-convex case argued informally.
- Update prediction assumes the centralised prediction is an unbiased estimator, which fails under client heterogeneity.
- Experiments use a fixed attacker that participates every round and only a small number of tasks.
Experimental setup
Datasets: UCI Census CIFAR-10 Reddit MNIST
Benchmarks: UCI Census CIFAR-10 Reddit
Metrics: accuracy change in fairness (accuracy disparity)
Baselines: Krum trimmed mean weak differential privacy unfair-update detection FedAdaGrad FedYogi FedAdam
Builds on: FedAvg model replacement attack Krum trimmed mean DP-FedAvg agnostic FL q-FFL
Closest papers by meaning
- To be Robust or to be Fair: Towards Fairness in Adversarial Training (ICML 2021 · similarity 0.90)
- On the Alignment between Fairness and Accuracy: from the Perspective of Adversarial Robustness (ICML 2025 · similarity 0.90)
- Adversarial Fairness Network (AAAI 2024 · similarity 0.88)
- DAFA: Distance-Aware Fair Adversarial Training (ICLR 2024 · similarity 0.88)
- Optimal Fair Learning Robust to Adversarial Distribution Shift (ICML 2025 · similarity 0.87)
- Towards Fairness-Aware Adversarial Learning (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.