Research map / Federated learning and privacy / Personalized and heterogeneous FL
Complementary Knowledge Distillation for Robust and Privacy-Preserving Model Serving in Vertical Federated Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Complementary Knowledge Distillation (CKD), which encodes only the active party's unlearned (complementary) label information via a LogitBoost-based Complementary Label Coding and distills it passive-to-active and passive-to-passive, to make VFL model serving robust to arbitrarily-aligned data while limiting label leakage.
Problem
Model serving in vertical federated learning must remain accurate when samples are arbitrarily aligned across passive parties (including stragglers) while preventing passive parties from inferring labels from their bottom-model outputs.
Approach
Instead of distilling the full federated teacher's knowledge to passive parties, the active party computes a Complementary Label Coding (shown equivalent to LogitBoost pseudo-residuals) so passive parties learn only label information the local model has not yet captured; a simplex-weighted aggregator then combines available passive bottom models, and knowledge is distilled both passive-to-active a
Key results
- On Criteo, CKD reached 74.5% partially-aligned AUC (P-Util) with privacy leakage AUC of 59.7%, versus 73.2% leakage for VFMD (ensemble distillation baseline).
- On Avazu, CKD achieved 74.3% P-Util with 57.9% leakage AUC, while SplitKD had 70.2% P-Util and VFMD had 72.5% leakage.
- With the simplex layer on Criteo, AUC rose from 74.3 (1 aligned passive party) to 76.6 (4 aligned), above the averaging layer's 76.2 at 4 parties.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CKD instead of SplitKD | VFL model serving · Criteo | AUC (L-Util, %): 72.6 -> 72.5 | No clear effect |
| Use CKD instead of VFMD | VFL model serving · Criteo | AUC (P-Util, %): 74.3 -> 74.5 | Helped |
| Use CKD instead of PtyDrop | VFL model serving · Criteo | AUC (privacy leakage, %): 62.4 -> 59.7 | Helped |
| Use CKD instead of VFMD | VFL model serving · Avazu | AUC (P-Util, %): 74.2 -> 74.3 | Helped |
| Use CKD instead of PtyDrop | VFL model serving · Avazu | AUC (privacy leakage, %): 60.4 -> 57.9 | Helped |
| Use CKD instead of VFMD | VFL model serving · HetRec | AUC (P-Util, %): 73.1 -> 73.4 | Helped |
| Use CKD instead of PtyDrop | VFL model serving · HetRec | AUC (privacy leakage, %): 61.6 -> 60.6 | Helped |
| Use CKD instead of VFMD | VFL model serving · MIMIC-III | AUC (P-Util, %): 69.2 -> 69.3 | Helped |
| Use CKD instead of PtyDrop | VFL model serving · MIMIC-III | AUC (privacy leakage, %): 57.3 -> 56.7 | Helped |
| Add p2a distillation loss (ablation) | CKD · Criteo | AUC (L-Util, %): 72.1 -> 72.5 | Helped |
| Add p2a distillation loss (ablation) | CKD · Criteo | AUC (privacy leakage, %): 59.6 -> 59.4 | Helped |
| Add p2p distillation loss (ablation) | CKD with p2a loss · Avazu | AUC (P-Util, %): 74.2 -> 74.3 | Helped |
Surprising findings
- Adding passive-to-passive distillation slightly increased label leakage while improving utility, a trade-off the authors attribute to knowledge sharing among passive parties.
- PtyDrop yields lower local utility than the standalone Local baseline on unaligned data.
Limitations
- Semi-honest threat model only; Byzantine attackers and collusion are not considered.
- Privacy guarantee (Theorem 1) holds only as the local model's standard error approaches zero.
- Cryptographic methods were excluded from comparison due to cost, so the privacy-efficiency trade-off against them is not measured.
Experimental setup
Datasets: Criteo Avazu HetRec MIMIC-III
Benchmarks: Criteo Avazu HetRec MIMIC-III
Metrics: AUC FL-AUC privacy leakage AUC (PMC attack)
Baselines: Local Vanilla VFL VFEns PtyDrop SplitKD VFMD
Builds on: LogitBoost knowledge distillation FedDF ensemble distillation party-wise dropout SplitKD
Closest papers by meaning
- Federated Transformer: Multi-Party Vertical Federated Learning on Practical Fuzzily Linked Data (NeurIPS 2024 · similarity 0.87)
- Federated Learning via Input-Output Collaborative Distillation (AAAI 2024 · similarity 0.87)
- Little Is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learning (AAAI 2025 · similarity 0.86)
- Towards the Robustness of Differentially Private Federated Learning (AAAI 2024 · similarity 0.86)
- PubSub-VFL: Towards Efficient Two-Party Split Learning in Heterogeneous Environments via Publisher/Subscriber Architecture (NeurIPS 2025 · similarity 0.86)
- Noise-Aware Algorithm for Heterogeneous Differentially Private Federated Learning (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.