Research map / Adversarial robustness and security / Black-box and transfer attacks
Black-Box Adversarial Attacks on LLM-Based Code Completion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes INSEC, a black-box attack that injects an optimized adversarial comment into code completion queries to make LLM-based completion engines (including GitHub Copilot and GPT-3.5-Turbo-Instruct) generate more insecure code while preserving functional correctness.
Problem
LLM-based code completion engines can be induced to generate insecure code, and existing attacks require white-box access to weights or training data, which is unrealistic for deployed black-box services like GitHub Copilot.
Approach
Manipulate only the input: insert a short, fixed adversarial attack string as a comment on the line above the completion point, optimized via query-based random search (mutation and top-n selection) with six initialization strategies (e.g., security-critical token, inversion, sanitizer), using a proxy tokenizer so no model internals are needed.
Key results
- INSEC increases vulnerable completion rate by up to 60% absolute across evaluated engines, with less than 22% relative decrease in functional correctness.
- On GPT-3.5-Turbo-Instruct, average vulnerability rate rises from 22% to 76% (or 73% with optimization on that model) while functional correctness is virtually unchanged.
- Total optimization cost is about $5.80 per CWE for GPT-3.5-Turbo-Instruct and an estimated $6 to $12 for open-weight models; the attack also succeeds against GitHub Copilot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Insert an optimized attack comment | GPT-3.5-Turbo-Instruct · 16 CWEs | vulnerability rate: 22% -> 73% | Helped |
| Apply an attack string optimized on StarCoder-3B | GPT-3.5-Turbo-Instruct · 16 CWEs | vulnerability rate: 22% to 55% | Helped |
| Apply an attack string optimized on CodeLlama-7B | GPT-3.5-Turbo-Instruct · 16 CWEs | vulnerability rate: 22% to 59% | Helped |
| Optimize the initialized attack string (ablation) | GPT-3.5-Turbo-Instruct · CWE-022 Path Traversal | vulnerability rate: 65% -> 96% | Helped |
| Optimize a randomly initialized attack string (ablation) | StarCoder-3B · CWE-131 Buffer Size | vulnerability rate: ≤ 14% -> 63% | Helped |
| Place the attack string as a comment on the line above the completion (ablation) | INSEC · Validation datasets | vulnerability rate and functional correctness | Helped |
| Format the attack string as a comment rather than inserting it without a comment symbol (ablation) | INSEC · Validation datasets | vulnerability rate: +6% | Helped |
| Format the attack string as a comment rather than inserting it without a comment symbol (ablation) | INSEC · Validation datasets | functional correctness: +11% | Helped |
| Use initialization strategies without optimization (ablation) | INSEC · Validation datasets | vulnerability rate: 50% | Helped |
| Optimize initialized attack strings rather than stopping after initialization (ablation) | INSEC · Validation datasets | vulnerability rate | Helped |
| Set attack-string pool size to 20 rather than too small or too large (ablation) | INSEC on StarCoder-3B · Validation datasets | attack impact | Helped |
| Increase attack-string length from one token to five tokens (ablation) | INSEC on StarCoder-3B · Validation datasets | vulnerability rate | Helped |
Surprising findings
- More capable engines (GPT-3.5-Turbo-Instruct, Copilot, larger StarCoder2) suffer less functional-correctness loss, suggesting future stronger models may be even more vulnerable.
- Attack strings optimized for one model partially transfer to other models, raising vulnerability on GPT-3.5-Turbo-Instruct from 22% to 55-59%.
Limitations
- Attack reduces functional correctness for some weaker engines (e.g., StarCoder-3B around 75% ratio at pass@1), with the authors noting stronger attacks could explicitly preserve correctness.
- Evaluation covers a limited set of CWEs and single-CWE attacks; multi-CWE composition is only explored for GPT-3.5-Turbo-Instruct.
- Final attack strings are withheld for ethical reasons, limiting direct reproducibility of the strongest attacks.
Experimental setup
Datasets: custom vulnerability dataset (16 CWEs, 5 languages, 12 tasks per CWE) HumanEval-derived multilingual completion dataset (GPT-4 reference solutions) HumanEval-X (validation of reference solutions) RepoBench
Benchmarks: custom INSEC vulnerability test set (16 CWEs) HumanEval-based multilingual functional correctness RepoBench
Metrics: vulnerability rate (vulRate) pass@k functional correctness ratio exact match edit similarity CodeBLEU
Baselines: unattacked engine (original completion) initialization-only attack strings CodeQwen tokenizer proxy variants
Builds on: fill-in-the-middle code completion random optimization jailbreak attacks Asleep at the Keyboard (insecure code evaluation)
Closest papers by meaning
- Security Attacks on LLM-based Code Completion Tools (AAAI 2025 · similarity 0.89)
- CodeMimicry: Exploiting Safety Generalization Lag in Large Language Models via Structured Code Completion (NeurIPS 2026 · similarity 0.87)
- FloatDoor: Platform triggered Backdoors in LLMs (NeurIPS 2026 · similarity 0.86)
- BaxBench: Can LLMs Generate Correct and Secure Backends? (ICML 2025 · similarity 0.86)
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.86)
- ProSec: Fortifying Code LLMs with Proactive Security Alignment (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.