Research map / Multi-agent systems and games / LLM agents

Helmsman: Autonomous Synthesis of Federated Learning Systems via Collaborative LLM Agents

ICLR 2026 · Multi-agent systems and games › LLM agents · approach: Agent data synthesis and behavior studies

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Helmsman is a multi-agent LLM system that turns a federated learning task specification into a validated FL codebase via human-verified planning, supervised modular coding, and a closed-loop simulation-based debug cycle, evaluated on the new 16-task AgentFL-Bench.

Problem

Designing and deploying robust federated learning systems requires selecting, combining, and tuning many strategies for heterogeneity and system constraints, which is manual and brittle; existing code-generation agents struggle with this system-level complexity.

Approach

A three-phase multi-agent pipeline (interactive planning with self-reflection and human approval, supervisor-managed coder/tester teams per FL module, and a closed-loop Flower simulation with hierarchical L1/L2 diagnosis and automated debugging) that synthesizes complete FL systems from natural-language queries.

Key results

• On AgentFL-Bench, the Helmsman (Claude Sonnet 4.5 / GPT-5.1) pipeline reaches a 100% success rate across 16 tasks versus 37.5% for Codex and 43.75% for Claude Code.
• On Split-CIFAR100 continual learning (5 tasks), the Helmsman-generated solution reaches 51.04% accuracy vs 34.89% for the best baseline (TARGET).
• Helmsman's generated strategy beats FedAvg and FedProx on most heterogeneity tasks (e.g., Q5 HAR 96.28 vs 94.84 FedAvg) but trails some specialized methods (e.g., Q1 76.71 vs 78.96 for specialized).

Surprising findings

• The GPT-5.1 backend was the cheapest configuration for Helmsman while also achieving 100% success, despite lower-cost models often being assumed weaker.
• Removing dual-layer verification reduced API cost but caused failures on all tasks.

Limitations

• Success is judged largely by whether the pipeline completes and passes internal checks; generated solutions are sometimes below specialized baselines (e.g., Q1, Q8, Q15).
• Convergence is not guaranteed; the debugging loop is capped at 10 attempts and may halt on complex tasks.
• Relies on proprietary LLM APIs, so reproducibility depends on model access.

Experimental setup

Datasets: CIFAR-10-LT CIFAR-100-C CIFAR-10N Office-Home UCI-HAR Speech Commands Fed-ISIC2019 Caltech101 CIFAR-100 FEMNIST CIFAR-10 DermaMNIST

Benchmarks: AgentFL-Bench

Metrics: test accuracy forgetting success rate API cost total tokens walltime

Baselines: FedAvg FedProx FedNova FedNS HeteroFL FedPer FAST FedWeIT FedEWC FedLwF TARGET Codex (GPT-5.1-Codex)

Builds on: LangGraph Flower ReAct CodeSim AgentCoder

Closest papers by meaning

• Federated Learning with Heterogeneous LLMs: Integrating Small Student Client Models with a Large Hungry Model (AAAI 2025 · similarity 0.86)
• MAS-GPT: Training LLMs to Build LLM-based Multi-Agent Systems (ICML 2025 · similarity 0.85)
• FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents (ICML 2026 · similarity 0.85)
• ComfyBench: Benchmarking LLM-based Agents in ComfyUI for Autonomously Designing Collaborative AI Systems (CVPR 2025 · similarity 0.85)
• AgentSwift: Efficient LLM Agent Design via Value-Guided Hierarchical Search (AAAI 2026 · similarity 0.85)
• How to Train Your LLM Web Agent: A Statistical Diagnosis (NeurIPS 2025 · similarity 0.85)