Research map / Adversarial robustness and security / Backdoor and poisoning

Concept-ROT: Poisoning Concepts in Large Language Models with Model Editing

ICLR 2025 · Adversarial robustness and security › Backdoor and poisoning · approach: Backdoors in large multimodal models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Concept-ROT, which uses a single ROME-style rank-one edit keyed on a concept vector (from representation reading) to insert trojans that jailbreak safety-tuned LLMs when a high-level concept is present, with ROT (token-trigger variant) also evaluated on HarmBench.

Problem

Existing trojan attacks on LLMs require large fine-tuning data or are limited to fixed token triggers; model editing trojans have only linked single words to fixed outputs. The paper asks whether rank-one model editing can insert trojans with concept-level triggers and complex behaviors such as jail

Approach

Use concept vectors extracted via representation reading (difference of activations with PCA) as the ROME key, and optimize the value vector to produce adversarial target outputs, so that prompts whose activations contain the concept trigger the behavior with a single layer edit and no benign control data.

Key results

• Concept-ROT (no control data) achieves 94.8% ASR on Gemma-7B and 87.9% on Llama-3.1-8B concept triggers with Open-LLM scores essentially unchanged (53.4 vs 53.5 and 69.8 vs 69.6), in about 15 s.
• ROT with the 'tartan' token trigger reaches 78.0% (Gemma-7B), 93.1% (Llama-3.1-8B) HarmBench ASR with direct-request ASR nearly unchanged (8.18% vs 9.43%), while FT and LoRA have high ASR even without the trigger.
• ROT achieves high ASR with as few as 5 harmful examples on most models, with Zephyr-7B+AT as the exception.

Measured effects reported in the paper

Surprising findings

• FT and LoRA trojans have high attack success rates even without the trigger, indicating poor stealthiness, whereas the edit remains selective.
• Concept-ROT struggled to find effective concept representations for Mistral-7B-v2, and LA had very low ASR on that model despite 100% train-set ASR.

Limitations

• Concept vectors are imperfect, causing false negatives and positives where on- and off-concept score distributions overlap; Mistral-7B-v2 concept vectors were less accurate.
• Which concepts or layers will serve as effective triggers cannot be determined a priori.
• Model editing trojans may be detectable by weight analysis methods; only single-layer edits were studied.

Experimental setup

Datasets: HarmBench standard subset Open-LLM benchmarks CounterFact Safe-RLHF SQuAD 2.0 Synthetic concept dataset (Llama-3.1-8B-IT generated)

Benchmarks: HarmBench Open-LLM (ARC-c, HellaSwag, TruthfulQA, MMLU, Winogrande, GSM8K) CounterFact Synthetic concept dataset

Metrics: attack success rate Open-LLM average score runtime false positive rate

Baselines: Fine-tuning (FT) LoRA Logit Anchoring Layerwise Weight Poisoning GCG AutoDAN Prefilling attack

Builds on: ROME BadEdit Linear Artificial Tomography LoRA

Closest papers by meaning

• Stealth edits to large language models (NeurIPS 2024 · similarity 0.89)
• Position: Editing Large Language Models Poses Serious Safety Risks (ICML 2025 · similarity 0.88)
• BadEdit: Backdooring Large Language Models by Model Editing (ICLR 2024 · similarity 0.86)
• BackdoorLLM: A Comprehensive Benchmark for Backdoor Attacks and Defenses on Large Language Models (NeurIPS 2025 · similarity 0.85)
• Uncovering Safety Risks of Large Language Models through Concept Activation Vector (NeurIPS 2024 · similarity 0.85)
• Protecting Model Adaptation from Trojans in the Unlabeled Data (AAAI 2025 · similarity 0.85)