Research map / Federated learning and privacy / Personalized and heterogeneous FL

Federated Black-Box Adaptation for Semantic Segmentation

NeurIPS 2024 · Federated learning and privacy › Personalized and heterogeneous FL · approach: Federated learning with pretrained models · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BlackFed, a black-box federated semantic segmentation method using split networks where clients are updated with zeroth-order SPSA-GC and the server with first-order AdamW, plus per-client server checkpoints to mitigate forgetting.

Problem

Federated semantic segmentation exposes model architecture and gradients between clients and server, enabling training-data reconstruction attacks; the paper seeks FL for segmentation without sharing gradients or client model information.

Approach

Formulate FL as split neural network training where the client (a two-layer conv net) is optimized by zeroth-order SPSA-GC using only forward passes, while the DeepLabv3 server is optimized with first-order AdamW; the server stores per-client weight snapshots (BlackFed v2) to counter catastrophic forgetting.

Key results

• On Cityscapes (18 clients), BlackFed v2 reaches 0.75 local and 0.74 OOD mIoU versus 0.50 and 0.44 for individual training, close to white-box training (0.75 / 0.75).
• On CAMVID, BlackFed v2 averages 0.72 mIoU across clients versus 0.67 for BlackFed v1 and matches combined training trends across DeepLabv3, SegFormer and UNext.
• BlackFed v2 improves OOD mIoU on ISIC and most PolypGen centers over individual training, but gains on PolypGen C5 and C6 are limited.

Measured effects reported in the paper

Surprising findings

• Optimizing the server before the client gives unstable SPSA-GC results, so the client-first ordering was adopted.
• Increasing server or client epochs did not significantly change average performance in the hyperparameter study.

Limitations

• Client-side zeroth-order optimization is less stable and forgetting remains on some PolypGen centers (C5, C6).
• Server must store per-client weight snapshots, and BlackFed v2 requires hashmap storage at the server.
• Evaluated only on four segmentation datasets with small per-client data; no formal theory on convergence.

Experimental setup

Datasets: Cityscapes CAMVID ISIC 2016 ISIC 2017 ISIC 2018 PolypGen

Benchmarks: Cityscapes CAMVID ISIC PolypGen

Metrics: mIoU p-value

Baselines: Individual training Combined training White-box training FedAvg FedSeg FedPer BlackFed v1

Builds on: SPSA-GC split neural networks FedAvg DeepLabv3

Closest papers by meaning

• Data-Free Black-Box Federated Learning via Zeroth-Order Gradient Estimation (AAAI 2025 · similarity 0.88)
• FedST: Federated Style Transfer Learning for Non-IID Image Segmentation (AAAI 2024 · similarity 0.87)
• Personalized Federated Learning via Feature Distribution Adaptation (NeurIPS 2024 · similarity 0.85)
• Federated Learning via Input-Output Collaborative Distillation (AAAI 2024 · similarity 0.85)
• FedQS: Optimizing Gradient and Model Aggregation for Semi-Asynchronous Federated Learning (NeurIPS 2025 · similarity 0.85)
• F^3OCUS - Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristics (CVPR 2025 · similarity 0.85)