Research map / Federated learning and privacy / Personalized and heterogeneous FL
Federated Transformer: Multi-Party Vertical Federated Learning on Practical Fuzzily Linked Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Federated Transformer (FeT), a multi-party vertical federated learning framework for fuzzily linked identifiers, combining per-party transformer encoders with dynamic masking, party dropout, positional encoding averaging, and SplitAvg (DP plus MPC aggregation).
Problem
Vertical federated learning where multiple parties hold different features of the same instances but link records only through fuzzy identifiers, which degrades accuracy and raises privacy and communication costs as the number of parties grows.
Approach
Encode fuzzy identifiers into representations via multi-dimensional positional encoding inside a distributed transformer, learn identifier-based dynamic key-padding masks to filter wrongly linked records, average per-party positional encodings under MPC, and protect secondary representations with norm clipping, distributed Gaussian noise aggregated via MPC, and secondary subsampling for privacy am
Key results
- FeT outperforms baselines by up to 46% accuracy in 50-party fuzzy VFL settings according to the abstract.
- In two-party fuzzy VFL, FeT achieves RMSE 39.75 on house, 232.98 on bike, and 26.94 on hdb, beating FedSim (42.12, 235.67, 27.13).
- Dynamic masking improves accuracy by up to 13 percentage points, e.g., MNIST 85.47% with DM vs 72.89% without.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use FeT instead of FedSim | two-party fuzzy VFL · house | RMSE: 42.12 ± 0.23 -> 39.75 ± 0.29 | Helped |
| use FeT instead of FedSim | two-party fuzzy VFL · bike | RMSE: 235.67 ± 0.27 -> 232.98 ± 0.62 | Helped |
| use FeT instead of FedSim | two-party fuzzy VFL · hdb | RMSE: 27.13 ± 0.06 -> 26.94 ± 0.15 | Helped |
| add dynamic masking (ablation) | FeT · MNIST (50 parties) | accuracy: 72.89% ± 1.43% -> 85.47% ± 0.13% | Helped |
| add dynamic masking (ablation) | FeT · gisette (50 parties) | accuracy: 90.32% ± 0.52% -> 92.43% ± 0.24% | Helped |
| add dynamic masking (ablation) | FeT · hdb | RMSE: 29.13 ± 0.18 -> 26.94 ± 0.15 | Helped |
| add positional encoding (ablation) | FeT · house | RMSE: 43.28 ± 0.74 -> 41.34 ± 0.54 | Helped |
| add positional encoding (ablation) | FeT · bike | RMSE: 234.25 ± 0.93 -> 232.98 ± 0.62 | Helped |
| increase party dropout rate from 0 to 0.6 (ablation) | FeT · gisette (50 parties) | accuracy: 92.18% ± 0.47% -> 92.90% ± 0.41% | Helped |
| increase party dropout rate from 0 to 0.6 (ablation) | FeT · MNIST (50 parties) | accuracy: 83.54% ± 0.30% -> 85.36% ± 0.25% | Helped |
| set party dropout rate to 1.0 (ablation) | FeT · MNIST (50 parties) | accuracy: 85.36% ± 0.25% -> 61.21% ± 0.58% | Hurt |
| add positional encoding averaging (frequency 1 instead of 0) (ablation) | FeT · gisette (50 parties) | accuracy: 92.05% ± 0.15% -> 92.50% ± 0.19% | Helped |
Surprising findings
- Top1Sim achieves the highest accuracy under exact linkage, outperforming FeT, which is attributed to overfitting.
- MPC aggregation of 10,000-dimensional vectors adds only about one second of overhead with 100 parties.
Limitations
- Assumes common features exist across all parties.
- Strict privacy (low epsilon) can cause significant accuracy loss.
- Assumes correlation between identifiers and representations; performance may drop with randomly generated keys.
Experimental setup
Datasets: house bike hdb gisette MNIST
Benchmarks: house bike hdb gisette MNIST
Metrics: RMSE accuracy
Baselines: Solo Top1Sim FedSim SplitNN
Builds on: FedSim SplitNN DPSGD FedAvg
Closest papers by meaning
- PubSub-VFL: Towards Efficient Two-Party Split Learning in Heterogeneous Environments via Publisher/Subscriber Architecture (NeurIPS 2025 · similarity 0.89)
- Complementary Knowledge Distillation for Robust and Privacy-Preserving Model Serving in Vertical Federated Learning (AAAI 2024 · similarity 0.87)
- FedLMT: Tackling System Heterogeneity of Federated Learning via Low-Rank Model Training with Theoretical Guarantees (ICML 2024 · similarity 0.87)
- Federated Learning via Input-Output Collaborative Distillation (AAAI 2024 · similarity 0.87)
- FedWon: Triumphing Multi-domain Federated Learning Without Normalization (ICLR 2024 · similarity 0.86)
- A Bayesian Approach for Personalized Federated Learning in Heterogeneous Settings (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.