Research map / Large language models / Language, safety and interpretability
Evaluation of Large Language Models on Code Obfuscation (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Evaluates ChatGPT 3.5, Jurassic-2, and PaLM on detecting C++ code obfuscations and identifying code behavior across 21 base programs and many obfuscation types.
Problem
Whether LLMs can understand code behavior and detect functionality-preserving obfuscations, which tests code reasoning and analysis ability.
Approach
Systematic prompt-based evaluation of off-the-shelf LLMs on simple C++ programs under layered and single obfuscation transformations, with a strict accuracy criterion requiring both a correct answer and a consistent explanation.
Key results
- Jurassic-2 was most accurate at identifying code functionality, correct in 60% of responses, versus PaLM at 47% and ChatGPT 3.5 at 47.45%.
- Layered obfuscations were hardest, with only 20-40% of LLM responses correct.
- Prompt 3 (asking what the code does, without original code) gave the highest accuracy (~55%), while prompt 2 gave the lowest (~48%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Jurassic-2 instead of PaLM or ChatGPT 3.5 | LLMs identifying code functionality · All evaluated code responses | accuracy: 47% -> 60%; 47.45% -> 60% | Helped |
| Ask the model to describe the obfuscated code's functionality without the original code | All LLMs · Obfuscated-code functionality prompt | accuracy: ˜48% -> ˜55% | Helped |
| Include the original code when asking whether it has the same functionality as the obfuscated code | All LLMs · Code functionality comparison | accuracy: ˜55% -> ˜48% | Hurt |
| Layer multiple transformations in the obfuscation | All LLMs · Layered obfuscations | correct response rate: 20-40% | Hurt |
| Use identifier-changing obfuscations rather than code-insertion obfuscations | All LLMs · Obfuscated-code detection and behavior identification | accuracy | Helped |
| Add misleading documentation to the obfuscated code | All LLMs · Obfuscated-code behavior identification | accuracy | Hurt |
| Transform for-loops into recursive code | All LLMs · For-loop-to-recursion obfuscation | accuracy | No clear effect |
Surprising findings
- Including the original code in the prompt appeared to impair LLM understanding of the obfuscated code.
- LLMs were fooled by misleading documentation but not by recursion-replacing obfuscation of for-loops.
Limitations
- Only three LLMs evaluated with default API settings.
- Simple, short C++ programs; generalization to real-world obfuscated malware is untested.
- Accuracy difference across prompts (~7% range) may not be statistically significant.
Experimental setup
Datasets: 21 hand-written C++ base programs with multiple obfuscation variants
Benchmarks: self-constructed C++ obfuscation set (21 base codes)
Metrics: accuracy (correct answer with consistent explanation)
Baselines: ChatGPT 3.5 Jurassic-2 PaLM
Closest papers by meaning
- Can LLMs Obfuscate Code? A Systematic Analysis of Large Language Models into Assembly Code Obfuscation (AAAI 2025 · similarity 0.91)
- ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding (ICLR 2025 · similarity 0.89)
- Can LLM Replace Stack Overflow? A Study on Robustness and Reliability of Large Language Model Code Generation (AAAI 2024 · similarity 0.88)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.87)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.87)
- Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.