Research map / Large language models / Language, safety and interpretability
Security Attacks on LLM-based Code Completion Tools
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes jailbreaking and training-data-extraction attacks that exploit the workflow of LLM-based code completion tools (LCCTs) such as GitHub Copilot and Amazon Q, embedding harmful queries in filenames, cross-file calls, and code variables.
Problem
LLM-based code completion tools integrate multiple context sources, prioritize code over natural language, and rely on proprietary code training data, creating security risks (jailbreaking and training data extraction) that prior work on LCCTs (focused on code security and copyright) has overlooked.
Approach
Tailor jailbreak prompts to LCCT workflows: hide harmful queries in filenames (Filename Proxy), split across files (Cross-File), and in code variables/comments (Hierarchical Code Exploitation with Guided Trigger and Code Embedded levels), plus a code-driven attack that first leaks GitHub usernames via a URL prefix and then extracts emails and locations.
Key results
- Level 1 Guided Trigger Attack reaches 99.4% ASR on GitHub Copilot and 46.3% on Amazon Q, versus 18.8% for DAN on GPT-4 and 0% on GPT-4o.
- Privacy extraction from GitHub Copilot yields 2,173 valid GitHub usernames (80.36% accuracy), 54 exact-matching emails, and 314 matching locations (100 exact, 214 fuzzy).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a filename proxy attack | GitHub Copilot · Jailbreaking ASR micro-benchmark | ASR: 72.5% | Helped |
| Use a cross-file attack | GitHub Copilot · Jailbreaking ASR micro-benchmark | ASR: 52.3% | Helped |
| Use a Level 1 guided trigger attack | GitHub Copilot · Jailbreaking ASR micro-benchmark | ASR: 99.4% | Helped |
| Use a Level 1 guided trigger attack | Amazon Q · Jailbreaking ASR micro-benchmark | ASR: 46.3% | Helped |
| Add guiding words to the Level 1 attack (ablation) | GitHub Copilot · Level 1 jailbreaking | ASR: 7.50% (-91.9%) | Helped |
| Add guiding words to the Level 1 attack (ablation) | Amazon Q · Level 1 jailbreaking | ASR: 5.00% (-41.3%) | Helped |
| Embed sensitive words within a long sentence in the Level 2 attack (ablation) | GitHub Copilot · Level 2 jailbreaking | ASR: 28.8% (-12.5%) | Hurt |
| Embed sensitive words within a long sentence in the Level 2 attack (ablation) | Amazon Q · Level 2 jailbreaking | ASR: 15.0% (-7.3%) | Hurt |
| Embed sensitive words within a long sentence in the Level 2 attack (ablation) | GPT-3.5 · Level 2 jailbreaking | ASR: 17.5% (-16.3%) | Hurt |
| Embed sensitive words within a long sentence in the Level 2 attack (ablation) | GPT-4 · Level 2 jailbreaking | ASR: 15.0% (-1.3%) | Hurt |
| Embed sensitive words within a long sentence in the Level 2 attack (ablation) | GPT-4o · Level 2 jailbreaking | ASR: 40.0% (-1.3%) | Hurt |
| Use Go instead of Python (ablation) | Level 1 and Level 2 attacks on GitHub Copilot and Amazon Q · Jailbreaking | ASR: 98.8% (-0.6%); 71.3% (+25.0%); 50.6% (+9.3%); 31.9% (+ 9.6%) | Mixed |
Surprising findings
- Guided trigger attacks on general LLMs like GPT-4o achieve only 36.5% ASR, while DAN gets 0% on GPT-4o, showing code-based attacks transfer partially.
- Copilot's only observed output filter blocks '@' and '.' characters, which is easily circumvented.
Limitations
- Only 80 queries across four categories, with five trials; small benchmark scale.
- Attack adapted to LCCTs using pyautogui automation; ASR depends on closed-source tool versions and may change over time.
- Judge relies on GPT-4 with only preliminary human validation cited.
Experimental setup
Datasets: OpenAI user policy-violating queries generated by GPT-4 (80 queries across 4 categories) GitHub REST API user data for evaluation
Benchmarks: Custom jailbreak micro-benchmark (80 queries, 4 categories) GitHub REST API user data for privacy extraction
Metrics: Attack Success Rate (ASR) exact matching rate fuzzy matching rate
Baselines: DAN CodeAttack
Builds on: DAN CodeAttack Jailbroken
Closest papers by meaning
- Black-Box Adversarial Attacks on LLM-Based Code Completion (ICML 2025 · similarity 0.89)
- ProSec: Fortifying Code LLMs with Proactive Security Alignment (ICML 2025 · similarity 0.88)
- CodeMimicry: Exploiting Safety Generalization Lag in Large Language Models via Structured Code Completion (NeurIPS 2026 · similarity 0.88)
- h4rm3l: A Language for Composable Jailbreak Attack Synthesis (ICLR 2025 · similarity 0.87)
- Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs (NeurIPS 2024 · similarity 0.87)
- Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.