Research map / Adversarial robustness and security / Backdoor and poisoning

Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models

ICLR 2025 · Adversarial robustness and security › Backdoor and poisoning · approach: Backdoors in large multimodal models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes BEAT, a black-box input-level detector that flags triggered prompts for backdoor unalignment in LLMs by measuring how much concatenating an input distorts the sampled output distribution of a harmful probe (EMD over sampled refusals).

Problem

Defending LLM-as-a-service models against backdoor unalignment attacks, where a hidden trigger breaks safety alignment with sample-dependent malicious targets, and the defender only has black-box text access.

Approach

Discovers the probe concatenate effect: appending a triggered harmful prompt to a harmful probe sharply reduces the probe's refusal rate, while benign or untriggered inputs do not. BEAT detects triggers by measuring EMD between multi-sample output sets of the probe alone versus probe+input, using only the first ~10 tokens to capture refusal signal.

Key results

• BEAT achieves average AUROC 99.7% (SFT attacks, Llama/Mistral/GPT-3.5) and 99.6% (RLHF Trojan attacks) with TPR@FPR5% of 100%, while baselines average below 90% AUROC.
• BEAT detects jailbreak attacks (GCG universal, GCG non-universal, ICA) with AUROC 96.95%, 90.97%, and 98.82% respectively on Vicuna-7b-v1.5.
• Against a syntactic trigger, BEAT reaches 93.7% AUROC with one probe and 96.5% with three probes.

Measured effects reported in the paper

Surprising findings

• Adaptive attack 1, which trains the trigger to still cause refusal on the defender's probe, drops BEAT's AUROC to 42.58%.

Limitations

• Adaptive attacks that embed the trigger to keep refusing the specific probe can substantially reduce detection (AUROC 42.58%).
• Performance depends on the choice of malicious probe and sampling/temperature hyperparameters.
• Targeted jailbreaks that interact with the system including the detector are not covered.

Experimental setup

Datasets: AdvBench MaliciousInstruct UltraChat_200k one hundred malicious QA pairs (SFT backdoor data) RLHF Trojan competition models

Benchmarks: AdvBench MaliciousInstruct UltraChat_200k

Metrics: AUROC TPR@FPR5% inference seconds per sample

Baselines: ONION Deletion Paraphrase NLI distance variant Statistical KL variant

Builds on: ONION Deletion (backdoor sample detection) Paraphrase-based backdoor detection Trojan RLHF competition backdoors

Closest papers by meaning

• BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models (NeurIPS 2025 · similarity 0.91)
• Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models (AAAI 2025 · similarity 0.90)
• LT-Defense: Searching-free Backdoor Defense via Exploiting the Long-tailed Effect (NeurIPS 2024 · similarity 0.89)
• BackdoorAlign: Mitigating Fine-tuning based Jailbreak Attack with Backdoor Enhanced Safety Alignment (NeurIPS 2024 · similarity 0.89)
• Causality Based Front-door Defense Against Backdoor Attack on Language Models (ICML 2024 · similarity 0.89)
• CL-Attack: Textual Backdoor Attacks via Cross-Lingual Triggers (AAAI 2025 · similarity 0.89)