Research map / Large language models / Instruction and fine-tuning

MetaDefense: Defending Fine-tuning based Jailbreak Attack Before and During Generation

NeurIPS 2025 · Large language models › Instruction and fine-tuning · approach: Safety alignment after fine-tuning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MetaDefense, which trains an LLM to classify harmfulness of queries before generation and of partial responses during generation, defending against finetuning-based jailbreaks with unseen attack templates.

Problem

Finetuning-based jailbreak attacks (FJAttacks) that disguise harmful queries with attack templates unseen at alignment time bypass existing defenses, which fail to generalize across templates.

Approach

Reuse the same LLM via lightweight instruction tuning to predict "harmful" or "harmless" for the query (pre-generation) and for partial responses at adaptive intervals during decoding (mid-generation), enabling early termination without a separate guard model.

Key results

• On LLaMA-2-7B, MetaDefense averages 1.0% ASR under seen Direct Attack and 1.7%, 7.2%, 4.1% ASR under unseen Prefix Injection, Role Play, and Refusal Suppression, versus 3.5% (Booster) to 78.8% (Non-Aligned) baselines.
• On Qwen-2.5-3B-Instruct, MetaDefense averages 0.1% ASR on Direct Attack and 2.0%, 0.5%, 11.1% on unseen templates, matching LLM-Classifier-level robustness with a single model.
• Under 100-epoch finetuning, MetaDefense keeps ASR at 2.4% while baselines rise to 66-75%.

Measured effects reported in the paper

Surprising findings

• LLM-Classifier trained on the alignment data achieves near-perfect ASR across unseen templates, suggesting LLMs internally separate harmful queries even when existing defenses fail.
• Pre-generation defense alone outperforms mid-generation defense alone (e.g., Direct Attack ASR 1.4% vs 25.1% on LLaMA-2-7B).

Limitations

• Mid-generation defense alone yields high ASR, and false negatives on harmful queries remain (e.g., Role Play 6.5-8.0%).
• Evaluated on only three benign tasks and small models (3B-7B) with LoRA finetuning.
• Adaptive attacks are only partly tested (mislabelled prompt attack on Qwen only).

Experimental setup

Datasets: Alpaca SST2 AGNews GSM8K harmful query datasets used for alignment (cited source) MMLU High School Chemistry

Benchmarks: SST2 AGNews GSM8K MMLU High School Chemistry

Metrics: Attack Success Rate Finetune Testing Accuracy inference time memory

Baselines: Non-Aligned SFT RepNoise Vaccine Booster BackdoorAlign PTST LLM-Classifier Booster + LLaMA-Guard

Builds on: Booster Vaccine RepNoise BackdoorAlign LLaMA-Guard

Closest papers by meaning

• MT-JailBench: A Modular Benchmark for Understanding Multi-Turn Jailbreak Attacks (NeurIPS 2026 · similarity 0.90)
• Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs (NeurIPS 2025 · similarity 0.90)
• Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions (ICML 2025 · similarity 0.89)
• Systematic Scaling Analysis of Jailbreak Attacks in Large Language Models (NeurIPS 2026 · similarity 0.89)
• Cannot See the Forest for the Trees: Invoking Heuristics and Biases to Elicit Irrational Choices of LLMs (ICML 2025 · similarity 0.89)
• Metis: Learning to Jailbreak LLMs via Self-Evolving Metacognitive Policy Optimization (ICML 2026 · similarity 0.89)