Research map / Adversarial robustness and security / Backdoor and poisoning

BadJudge: Backdoor Vulnerabilities of LLM-As-A-Judge

ICLR 2025 · Adversarial robustness and security › Backdoor and poisoning · approach: Backdoors in large multimodal models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes backdoor attacks on LLM-as-a-Judge evaluators where adversary controls both candidate and evaluator, showing a single-token trigger poisoning 1% of evaluator data inflates scores, and that model merging removes the backdoor.

Problem

LLM-as-a-Judge evaluators can be backdoored by an adversary who controls both the candidate model and the evaluator training data, unfairly inflating the adversary's scores while misleading benign users.

Approach

Formalizes a chain-reaction backdoor where the candidate model emits a trigger token that activates a poisoned evaluator to give top scores, categorizes attacker data access into web poisoning, malicious annotator, and weight poisoning settings, and defends with linear model merging of two poisoned models.

Key results

• With 1% rare-word poisoning of the Mistral-7B evaluator, the adversary's mean score rises from about 1.4 to 4.6 out of 5 (tripling), with 81.2% ASR.
• Under the weight poisoning (full access) setting, the adversary's score goes from 1.5/5 to 4.9/5 with 93.8% ASR.
• Linear model merging reduces ASR from 93.8% to 0% on the full-assumptions point-wise attack while keeping clean accuracy near 53.8%.

Measured effects reported in the paper

Surprising findings

• Even under the weakest web-poisoning assumption, the adversary still achieves about a 20% score inflation.
• Label-rationale inconsistency detection could not distinguish poisoned evaluations, emphasizing stealthiness of the attack.

Limitations

• Continued fine-tuning defense requires clean data, a strong assumption.
• Trigger designs such as rare words are obvious, while stealthier syntactic and stylistic triggers have lower clean accuracy and effectiveness trade-offs.
• Experiments limited to 7B/8B-scale evaluators and a small set of tasks; claims of real-world threat are partly simulated.

Experimental setup

Datasets: Feedback-Collection Preference-Collection Ultrachat-200k MSMarco passages MT-Bench

Benchmarks: Feedback-Collection Preference-Collection MT-Bench MSMarco passages

Metrics: attack success rate clean accuracy mean score Hit@1 MRR@10

Baselines: In-context learning defense Continuous fine-tuning defense ONION BKI Chain-of-Scrutiny-style inconsistency detection

Builds on: Feedback-Collection model merging Backdoor Keyword Identification ONION

Closest papers by meaning

• Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples (NeurIPS 2026 · similarity 0.88)
• Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs (NeurIPS 2026 · similarity 0.87)
• BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models (NeurIPS 2025 · similarity 0.87)
• Probe before You Talk: Towards Black-box Defense against Backdoor Unalignment for Large Language Models (ICLR 2025 · similarity 0.86)
• Simulate and Eliminate: Revoke Backdoors for Generative Large Language Models (AAAI 2025 · similarity 0.86)
• BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models (ICLR 2024 · similarity 0.86)