Research map / Large language models / Language, safety and interpretability
ChatGPT-Generated Code Assignment Detection Using Perplexity of Large Language Models (Student Abstract)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CGCode Detector, a zero-shot detector for ChatGPT-generated code that scores code by perplexity, line-level perplexity variation, and burstiness after targeted masking perturbations refilled by a fine-tuned CodeBERT.
Problem
Detecting ChatGPT-generated programming assignments to protect academic integrity, a setting where existing text detectors were designed for natural language rather than code.
Approach
Adapts DetectGPT-style perturbation to code: masks high-perplexity lines more heavily, refills masks with a fine-tuned CodeBERT, and scores the original code by overall perplexity, cross-line perplexity deviation, and burstiness, where model-generated code ranks stably low.
Key results
- CGCode Detector achieves average AUC of 0.87 versus 0.56 for the GPTZero baseline, per the abstract.
- On C, CGCode Detector reaches AUC 0.95 with FPR 0.16 and FNR 0.08, against GPTZero's 0.90 AUC on C.
- On Python, CGCode Detector reaches AUC 0.92 versus 0.59 for GPTZero.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CGCode Detector instead of GPTZero | Code detection · C | AUC: 0.90 -> 0.95 | Helped |
| Use CGCode Detector instead of GPTZero | Code detection · C++ | AUC: 0.83 -> 0.88 | Helped |
| Use CGCode Detector instead of Writer | Code detection · C# | AUC: 0.52 -> 0.86 | Helped |
| Use CGCode Detector instead of Writer | Code detection · Java | AUC: 0.54 -> 0.82 | Helped |
| Use CGCode Detector instead of Writer | Code detection · JavaScript | AUC: 0.56 -> 0.81 | Helped |
| Use CGCode Detector instead of GPTZero | Code detection · Python | AUC: 0.59 -> 0.92 | Helped |
| Use CGCode Detector instead of GPTZero | Code detection · Average across programming languages | AUC: 0.56 to 0.87 | Helped |
Surprising findings
- Existing text detectors such as DetectGPT and RoBERTa-QA perform near chance on code, with AUC around 0.4 to 0.7 across languages.
Limitations
- Evaluation limited to one generator (text-davinci-003) in the abstract emphasis, and the paper reports results against text-davinci-003-generated code
- Results rely on a per-detector best-performance threshold choice
- Only six programming languages and a student-submission-like dataset from CodeNet
Experimental setup
Datasets: IBM CodeNet CGCode
Benchmarks: CGCode
Metrics: AUC FPR FNR
Baselines: GPT2-Detector DetectGPT RoBERTa-QA GPTZero Writer
Builds on: DetectGPT CodeBERT GPTZero
Closest papers by meaning
- Detecting AI-Generated Code Assignments Using Perplexity of Large Language Models (AAAI 2024 · similarity 0.95)
- Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting (AAAI 2025 · similarity 0.87)
- ObscuraCoder: Powering Efficient Code LM Pre-Training Via Obfuscation Grounding (ICLR 2025 · similarity 0.87)
- DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text (ICLR 2024 · similarity 0.86)
- Hot or Cold? Adaptive Temperature Sampling for Code Generation with Large Language Models (AAAI 2024 · similarity 0.86)
- Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.