Research map / Large language models / Language, safety and interpretability

CodeMimicry: Exploiting Safety Generalization Lag in Large Language Models via Structured Code Completion

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Safety alignment of LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CodeMimicry, a black-box jailbreak that wraps harmful requests in object-oriented code stubs for completion, refined over up to 5 feedback turns, reaching 96.25% ASR on AdvBench across 8 commercial/open LLMs.

Problem

Safety alignment trained mostly on natural language may not transfer to the code domain, leaving black-box LLMs vulnerable to jailbreaks that hide malicious intent in code completion requests.

Approach

Decompose a harmful query into a class/method stub with docstrings and comments describing the harmful functionality, wrap it in a benign code-completion trigger, and iteratively refine it using an attacker LLM and an LLM judge's feedback history; the paper also analyzes refusal-direction projections and activation steering in latent space.

Key results

• CodeMimicry achieves 96.25% ASR on AdvBench across 8 target LLMs with 1.51 average queries, versus 85.00% for the best template baseline (CodeChameleon) and 43.47% for the best automated baseline on HarmBench (AutoDan-Turbo).
• On HarmBench subset, CodeMimicry averages 96.07% ASR and 74.29% against Claude Sonnet 4 where baselines like PAIR and DRA reach 0%.
• Regex comment/docstring sanitizer reduces CodeMimicry ASR on GPT-4o from 90% to 20%, but it retains 20-26% ASR on the three tested models.

Surprising findings

• Successful jailbreak code representations stay close to benign code along the refusal direction, while failed attempts cluster as refusals in Llama-3-8B-Instruct.
• Template baselines like CodeChameleon and FlipAttack fluctuate widely across models, e.g., FlipAttack drops to 0% on Llama-3.1-70B.

Limitations

• Depends on target models having sufficient code and instruction-following capability; may degrade on small models.
• Relies on comments and docstrings as side channels; regex sanitization reduces ASR substantially though not to zero.
• Evaluation uses small subsets (50 AdvBench, 70 HarmBench) and a GPT-based judge with some disagreement against human experts.

Experimental setup

Datasets: AdvBench HarmBench

Benchmarks: AdvBench HarmBench

Metrics: attack success rate average queries syntax correctness rate runtime success rate functional validity rate

Baselines: CipherChat ReNeLLM CodeAttack FlipAttack EquaCode CodeChameleon PAIR DRA GPTfuzzer AutoDan-Turbo

Builds on: PAIR CodeChameleon CodeAttack Circuit Breaker

Closest papers by meaning

• Playing the Fool: Jailbreaking LLMs and Multimodal LLMs with Out-of-Distribution Strategy (CVPR 2025 · similarity 0.91)
• STAIR: Improving Safety Alignment with Introspective Reasoning (ICML 2025 · similarity 0.90)
• Safety Alignment Should be Made More Than Just a Few Tokens Deep (ICLR 2025 · similarity 0.90)
• Fail-Closed Alignment for Large Language Models (NeurIPS 2026 · similarity 0.90)
• h4rm3l: A Language for Composable Jailbreak Attack Synthesis (ICLR 2025 · similarity 0.90)
• Lifelong Safety Alignment for Language Models (NeurIPS 2025 · similarity 0.90)