Research map / Large language models / Agents, code and math
Can LLMs Obfuscate Code? A Systematic Analysis of Large Language Models into Assembly Code Obfuscation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds MetamorphASM Dataset (MAD) of 328,200 assembly snippet pairs with three obfuscations (dead code, register substitution, control flow change) and benchmarks LLMs on generating obfuscated x86 assembly using delta entropy, cosine similarity and human review.
Problem
Whether LLMs can generate new obfuscated assembly code, which could let malware rewrite itself via API calls to evade anti-virus engines, and how reliably current LLMs do this.
Approach
Creates a script-generated assembly obfuscation dataset with three techniques and evaluates zero-shot and few-shot LLM obfuscation using entropy-change and cosine-similarity proxies plus expert ranking, also fine-tuning CodeT5 on the data.
Key results
- GPT-4o-mini achieved about 26.9% delta entropy on dead code insertion at 0-shot with cosine similarity 0.93, and 47-49% delta entropy on control flow change with CS around 0.93-0.94 in few-shot settings.
- Human evaluation ranked GPT-4o-mini and DeepSeekCoder-v2 as the top two obfuscators across all three techniques.
- LLaMA 3.1 and CodeGemma/CodeLlama produced little or no effective obfuscation (delta entropy around 2% or less, with LLaMA 3.1 failing to output valid assembly at 10+ shots).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 1-shot instead of zero-shot prompting (ablation) | GPT-4o-mini · Dead Code Insertion | Delta entropy (%): 26.90 -> 21.00 | Hurt |
| Use 15-shot instead of zero-shot prompting (ablation) | DeepSeekCoder-v2 · Dead Code Insertion | Delta entropy (%): 19.50 -> 27.50 | Helped |
| Use 15-shot instead of zero-shot prompting (ablation) | Codestral · Dead Code Insertion | Delta entropy (%): 30.25 -> 11.93 | Hurt |
| Use 15-shot instead of zero-shot prompting (ablation) | DeepSeekCoder-v2 · Register substitution | Delta entropy (%): 15.71 -> 18.55 | Helped |
| Use 15-shot instead of zero-shot prompting (ablation) | Codestral · Register substitution | Delta entropy (%): 40.12 -> 41.33 | Mixed |
| Use 15-shot instead of zero-shot prompting (ablation) | GPT-4o-mini · Control Flow Change | Delta entropy (%): 15.61 -> 48.63 | Helped |
| Use 15-shot instead of zero-shot prompting (ablation) | DeepSeekCoder-v2 · Control Flow Change | Delta entropy (%): 49.20 -> 52.55 | Helped |
| Use 15-shot instead of zero-shot prompting (ablation) | CodeLlama · Dead Code Insertion | Delta entropy (%): 2.20 -> 1.55 | No clear effect |
| Fine-tune on MAD (ablation) | CodeT5 · Dead Code Insertion | Delta entropy (%): The paper reports 0.06 | Hurt |
| Fine-tune on MAD (ablation) | CodeT5 · Register substitution | Delta entropy (%): The paper reports 0.01 | Hurt |
Surprising findings
- General-purpose GPT models outperformed assembly-trained and code-specialized LLMs such as DeepSeekCoder-v2, Codestral and CodeLlama.
- CodeT5 fine-tuned on MAD produced very low entropy change, often merely reproducing the original code.
Limitations
- No functional correctness verification at scale; relies on entropy and cosine similarity proxies and a two-month manual review.
- Dataset uses 20-line snippets and removes JMP/CALL instructions, limiting realism.
- Thresholds for acceptable obfuscation were set by inspecting the same LLM outputs.
Experimental setup
Datasets: MetamorphASM Dataset (MAD)
Benchmarks: MetamorphASM Dataset (MAD)
Metrics: delta entropy cosine similarity human expert ranking
Baselines: GPT-3.5 Starcoder CodeGemma CodeLlama Codestral DeepSeekCoder-v2 LLaMA 3.1
Builds on: CodeT5
Closest papers by meaning
- Evaluation of Large Language Models on Code Obfuscation (Student Abstract) (AAAI 2024 · similarity 0.91)
- ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding (ICLR 2025 · similarity 0.86)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.86)
- Security Attacks on LLM-based Code Completion Tools (AAAI 2025 · similarity 0.86)
- BaxBench: Can LLMs Generate Correct and Secure Backends? (ICML 2025 · similarity 0.85)
- Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.