Research map / Federated learning and privacy / Personalized and heterogeneous FL
Recurrent Early Exits for Federated Learning with Heterogeneous Clients
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReeFL, a recurrent early-exit module shared across sub-models in federated learning that fuses multi-layer class tokens into one classifier and uses per-client best-exit self-distillation, evaluated on CIFAR-100, FEMNIST and SpeechCommands with DeiT-S under full and PEFT fine-tuning.
Problem
Federated learning with heterogeneous clients whose compute and memory budgets differ; existing early-exit approaches train multiple classifiers that compete, suffer from partial data at deeper exits, and rely on hand-picked teacher sub-models for distillation.
Approach
Replace multiple per-exit classifiers with a single lightweight transformer-based recurrent early-exit module shared across all sub-models, which fuses class tokens from multiple layers into one shared classifier and feeds modulated class tokens back into the backbone; the teacher exit for distillation is chosen dynamically per client by lowest running training CE loss.
Key results
- ReeFL with full fine-tuning on CIFAR-100 (alpha=1000, 4 exits) reaches 76.42% mean exit accuracy vs 67.29% for best baseline ExclusiveFL and 57.52% for DepthFL.
- With frozen backbone on SpeechCommands (4 exits), ReeFL achieves 66.09% vs 26.53% for InclusiveFL.
- ReeFL outperforms baselines in all but one scenario (SA, CIFAR-100 alpha=1000, 4 exits); communication cost is roughly 1 MB higher per round than DepthFL on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReeFL instead of the strongest listed baseline | DeiT-S with full fine-tuning and 4 exits · CIFAR-100 α=1000; α=1.0; α=0.1, FEMNIST, and SpeechCmds | accuracy | Helped |
| Use ReeFL instead of the strongest listed baseline | DeiT-S with LoRA and 4 exits · CIFAR-100 α=1000; α=1.0; α=0.1, FEMNIST, and SpeechCmds | accuracy | Helped |
| Add feature modulation (ablation) | ReeFL with a frozen backbone · CIFAR-100 α=1000, 4 exits | accuracy: 53.89±0.0 -> 67.52±0.1 | Helped |
| Add feature modulation (ablation) | ReeFL with a frozen backbone · SpeechCmds, 4 exits | accuracy: 20.14±0.21 -> 66.09±0.08 | Helped |
| Add feature modulation (ablation) | ReeFL with LoRA · CIFAR-100 α=1000, 4 exits | accuracy: 73.69±0.07 -> 73.9±0.03 | No clear effect |
| Inject Ree at every layer rather than only at exit layers (ablation) | ReeFL with a frozen backbone · CIFAR-100 α=1000, 4 exits | accuracy: -6.89 | Helped |
| Inject Ree at every layer rather than only at exit layers (ablation) | ReeFL with a frozen backbone · SpeechCmds, 4 exits | accuracy: -20.67 | Helped |
| Add ReeFL's best-training-loss teacher distillation (ablation) | ReeFL with a frozen backbone · CIFAR-100 α=1000, 4 exits | accuracy: 67.04±0.09 -> 67.52±0.1 | Helped |
| Add ReeFL's best-training-loss teacher distillation (ablation) | ReeFL with LoRA · CIFAR-100 α=1000, 4 exits | accuracy: 73.47±0.01 -> 73.9±0.03 | Helped |
Surprising findings
- Using a shared classifier without the Ree module reduces accuracy below DepthFL on CIFAR-100 alpha=1000 with LoRA (70.74 vs 71.22), showing the fusion module is key.
- Full fine-tuning with DepthFL or InclusiveFL underperforms PEFT on CIFAR-100 and FEMNIST, and DepthFL collapses to near-chance accuracy in several 12-exit PEFT settings.
Limitations
- Communication cost is slightly higher than DepthFL because the shared module and classifier are sent every round.
- Feature modulation requires uniform state size across layers, so it does not directly apply to architectures like ResNets.
- Evaluated only on image and speech classification with DeiT-S backbone; not tested on language or with differential privacy.
Experimental setup
Datasets: CIFAR-100 FEMNIST SpeechCommandV2 ImageNet
Benchmarks: CIFAR-100 FEMNIST SpeechCommands
Metrics: mean test accuracy across exits communication cost (MB) MACs peak training memory
Baselines: ExclusiveFL InclusiveFL ScaleFL DepthFL
Builds on: DepthFL InclusiveFL ScaleFL FedAvg FedDyn
Closest papers by meaning
- FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.88)
- Tackling Feature-Classifier Mismatch in Federated Learning via Prompt-Driven Feature Transformation (NeurIPS 2025 · similarity 0.88)
- FedWon: Triumphing Multi-domain Federated Learning Without Normalization (ICLR 2024 · similarity 0.87)
- pFedMxF: Personalized Federated Class-Incremental Learning with Mixture of Frequency Aggregation (CVPR 2025 · similarity 0.87)
- (FL)$^2$: Overcoming Few Labels in Federated Semi-Supervised Learning (NeurIPS 2024 · similarity 0.87)
- FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Data (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.