Research map / Large language models / Language, safety and interpretability
Detecting AI-Generated Code Assignments Using Perplexity of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes AIGCode Detector, a zero-shot detector that flags AI-generated code by perplexity-guided masking, CodeBERT mask-filling perturbations, and a composite perplexity/burstiness score.
Problem
Detecting whether student programming assignments were generated by LLMs such as ChatGPT/text-davinci-003, where existing text detectors perform poorly on code.
Approach
Extends DetectGPT-style perturbation to code: masks are allocated preferentially to high-perplexity lines, CodeBERT (fine-tuned per language) fills the masks, and the original code is ranked against perturbed versions using a score combining overall perplexity, std of line-level perplexity, and burstiness.
Key results
- AIGCode Detector averages AUC 0.87 on text-davinci-003 code vs. 0.56 for GPTZero, per the abstract; per-language AUC for AIGCode ranges 0.81-0.95 in Table 1.
- Under code-blending mutation, bypass rate is 0.75 for AIGCode Detector vs. 0.87 for GPTZero.
- Using GPT-NeoX as the perplexity model gives the best AUC in Table 2 (0.74-0.87 across languages), with larger models performing better.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GPTZero with AIGCode Detector | code-origin detection · C | AUC: 0.90 -> 0.95 | Helped |
| replace GPTZero with AIGCode Detector | code-origin detection · C++ | AUC: 0.83 -> 0.88 | Helped |
| replace Writer with AIGCode Detector | code-origin detection · C# | AUC: 0.52 -> 0.86 | Helped |
| replace Writer with AIGCode Detector | code-origin detection · Java | AUC: 0.54 -> 0.82 | Helped |
| replace Writer with AIGCode Detector | code-origin detection · JavaScript | AUC: 0.56 -> 0.81 | Helped |
| replace GPTZero with AIGCode Detector | code-origin detection · Python | AUC: 0.59 -> 0.92 | Helped |
| use GPT2-xl instead of text-davinci-003 for perplexity calculation (ablation) | AIGCode Detector · six programming languages | AUC: C: 0.79; C++: 0.75; C#: 0.73; Java: 0.72; JavaScript: 0.71; Python: 0.69 | Hurt |
| use GPT-J instead of text-davinci-003 for perplexity calculation (ablation) | AIGCode Detector · six programming languages | AUC: C: 0.82; C++: 0.76; C#: 0.76; Java: 0.77; JavaScript: 0.79; Python: 0.78 | Hurt |
| use GPT-NeoX instead of text-davinci-003 for perplexity calculation (ablation) | AIGCode Detector · six programming languages | AUC: C: 0.87; C++: 0.83; C#: 0.80; Java: 0.74; JavaScript: 0.75; Python: 0.83 | Hurt |
| apply sampling techniques with higher temperature and altered top-p | AI-generated C code · AIGCode dataset | bypass rate: 0.37 | Hurt |
| blend human-written and AI-generated code | AI-generated C code · AIGCode dataset | bypass rate: 0.75 | Hurt |
Surprising findings
- Code blending (mixing human-truncated code with AI completion) bypasses both detectors at high rates (0.75-0.87), and sampling-based variation also raises bypass rates.
- Detection is near chance for several baselines on some languages (e.g., DetectGPT AUC 0.42-0.56 across languages).
Limitations
- Relies on a single generator (text-davinci-003), so generalization to other code LLMs is untested.
- Evaluated on a balanced dataset; real-world class imbalance not studied.
- Requires code rewriting/perturbation with many samples, which is costly.
Experimental setup
Datasets: CodeNet AIGCode
Benchmarks: AIGCode
Metrics: AUC FPR FNR bypass rate
Baselines: GPT2-Detector DetectGPT RoBERTa-QA Writer AI Detector GPTZero
Builds on: DetectGPT CodeBERT GPTZero
Closest papers by meaning
- ChatGPT-Generated Code Assignment Detection Using Perplexity of Large Language Models (Student Abstract) (AAAI 2024 · similarity 0.95)
- Position: On the Possibilities of AI-Generated Text Detection (ICML 2024 · similarity 0.88)
- Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting (AAAI 2025 · similarity 0.87)
- DNA-DetectLLM: Unveiling AI-Generated Text via a DNA-Inspired Mutation-Repair Paradigm (NeurIPS 2025 · similarity 0.86)
- Few-Shot Detection of Machine-Generated Text using Style Representations (ICLR 2024 · similarity 0.85)
- ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.