Research map / Large language models / Instruction and fine-tuning
Emerging Safety Attack and Defense in Federated Instruction Tuning of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that malicious federated instruction-tuning clients can compromise LLM safety with automatically generated unaligned data, and proposes a server-side post-hoc defense that fine-tunes the aggregated LLM on generated safe and normal data.
Problem
Federated instruction tuning (FedIT) of LLMs can lose safety alignment when malicious clients train on unaligned harmful instruction-response data, and existing model-level FL defenses fail to detect such stealthy attacks.
Approach
A data-poisoning attack in which malicious clients train on harmful-instruction/harmful-response pairs generated automatically by an off-the-shelf LLM, plus a post-hoc server-side defense that fine-tunes the aggregated model on automatically generated aligned (harmful instruction, safe response) and normal data, avoiding model-level client comparison.
Key results
- Safety attack reduces FedAvg Rule safety from 82.88 to 49.81 and MD-Judge from 66.15 to 25.96 on LMSYS-Chat with Beavertails (no-attack vs attack).
- Best existing FL defenses add at most about 4 points of safety improvement under attack.
- Level 2 post-hoc defense raises WildChat+Beavertails MD-Judge to 82.12 from 12.31 under attack, with MT-1 roughly unchanged (4.33 vs 4.54).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train malicious clients on Beavertails unaligned data instead of aligned data | FedAvg on Llama2-7B · AdvBench, LMSYS-Chat | MD-Judge: 66.15 -> 25.96 | Hurt |
| train malicious clients on Beavertails unaligned data instead of aligned data | FedAvg on Llama2-7B · AdvBench, WildChat | MD-Judge: 43.27 -> 12.31 | Hurt |
| add Level 2 post-hoc defense using generated data | FedAvg attacked with Beavertails · AdvBench, LMSYS-Chat | MD-Judge: 25.96 -> 84.23 | Helped |
| add Level 2 post-hoc defense using generated data | FedAvg attacked with Beavertails · AdvBench, WildChat | MD-Judge: 12.31 -> 82.12 | Helped |
| use classical FL aggregation defenses rather than plain FedAvg under safety attack | FedAvg with Beavertails attack · AdvBench, LMSYS-Chat | MD-Judge: 25.96 -> 27.88 | No clear effect |
| add Level 2 defense after aggregation | FL baselines under LMSYS-Chat + Beavertails attack · AdvBench | MD-Judge: 25.96 -> 84.23 | Helped |
| use 400 Level 3 defense steps rather than other tested step counts (ablation) | Level 3 defense · four attack settings | MT-1 | Helped |
| increase Level 3 defense training steps across [100, 200, 300, 400, 500] (ablation) | Level 3 defense · four attack settings | MD-Judge: 41.73\%, 29.42\% | Helped |
| fine-tune a pretrained Llama2 on generated defense data alone instead of normal local training (ablation) | Local+Gen · generated-data fine-tuning | helpfulness | Hurt |
| add generated defense data to federated fine-tuning under attack (ablation) | FedAvg under attack · generated-data defense | safety | Helped |
| add generated defense data to federated fine-tuning under attack (ablation) | FedAvg under attack · generated-data defense | helpfulness | No clear effect |
| generate attack data with different off-the-shelf LLMs (ablation) | FedAvg safety attack · LMSYS-Chat | MD-Judge | Hurt |
Surprising findings
- Krum assigned zero aggregation weight to two benign clients while still giving weight to malicious ones, showing model-level defenses misidentify clients.
- Post-hoc defense on benign-only FedAvg also improved safety (e.g., Level 3 gave +26.35% MD-Judge on WildChat with no attack).
Limitations
- Only Llama2-7B is evaluated as base model.
- Defense relies on a fixed number of fine-tuning steps and generated data quality; Level 3 tuning steps affect helpfulness.
- Attack evaluated on simulated FL with a small number of clients and LoRA fine-tuning.
Experimental setup
Datasets: LMSYS-Chat WildChat Dromedary-verbose Wizard-evol Beavertails MaliciousGen CodeAlpaca
Benchmarks: AdvBench MT-Bench HumanEval
Metrics: Rule-based safety rate MD-Judge safety score reward model score MT-Bench first-turn score HumanEval pass@1
Baselines: FedAvg Median Trimmed mean Krum DnC FoolsGold Residual
Builds on: OpenFedLLM FedAvg LoRA Krum
Closest papers by meaning
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (ICLR 2024 · similarity 0.90)
- RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards (NeurIPS 2025 · similarity 0.88)
- Safety Layers in Aligned Large Language Models: The Key to LLM Security (ICLR 2025 · similarity 0.88)
- Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.87)
- Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuning Attack (ICML 2025 · similarity 0.87)
- Navigating the Safety Landscape: Measuring Risks in Finetuning Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.