Research map / Large language models / Language, safety and interpretability
Uncovering LLM-Generated Code: A Zero-Shot Synthetic Code Detector via Code Rewriting
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a zero-shot, black-box synthetic code detector that scores code by the average similarity between it and LLM-rewritten variants, using a GraphCodeBERT similarity model trained with unsupervised SimCSE.
Problem
Detecting LLM-generated (synthetic) code, where existing zero-shot text detectors based on token log probabilities perform poorly because code contains many low-entropy tokens.
Approach
LLM rewrites of synthetic code stay closer to the original than rewrites of human-written code, so the embedding similarity between a snippet and its LLM rewrites serves as a black-box detection score; the similarity model is trained with self-supervised SimCSE on unlabeled code.
Key results
- Ours (m=8, CodeLlama detector) reaches 80.16 average AUROC on APPS and 81.81 on MBPP, versus the best baseline LogRank at 65.38 and NPR at 60.82.
- The method with m=4 rewrites achieves 78.49 average AUROC on APPS and 77.99 on MBPP using CodeLlama as detector.
- On C++ (Code Contest), the m=8 method reaches 89.87 AUROC for CodeLlama-generated code, outperforming DetectGPT at 62.99.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use code rewriting with similarity measurement rather than in-fill perturbation with similarity (ablation) | Zero-shot synthetic code detection · APPS | AUROC: 74.55 -> 87.77; 71.55 -> 76.96; 74.23 -> 80.23; 53.43 -> 74.51 | Helped |
| Use similarity measurement rather than token-wise LogProb scoring (ablation) | Code rewriting detector · APPS | AUROC: 57.29 -> 87.77; 46.70 -> 76.96; 50.58 -> 80.23; 55.13 -> 74.51 | Helped |
| Use similarity measurement rather than token-wise LogRank scoring (ablation) | Code rewriting detector · APPS | AUROC: 75.28 -> 87.77; 64.76 -> 76.96; 75.85 -> 80.23; 67.69 -> 74.51 | Helped |
| Use similarity measurement rather than token-wise Rank scoring (ablation) | Code rewriting detector · APPS | AUROC: 62.34 -> 87.77; 55.42 -> 76.96; 60.59 -> 80.23; 59.34 -> 74.51 | Helped |
| Use similarity measurement rather than token-wise Entropy scoring (ablation) | Code rewriting detector · APPS | AUROC: 60.39 -> 87.77; 59.42 -> 76.96; 68.20 -> 80.23; 59.35 -> 74.51 | Helped |
| Use code rewriting rather than in-fill perturbation (ablation) | Similarity-based synthetic code detector · APPS (GPT-4-generated) | AUROC: 53.43 -> 74.51 | Helped |
| Use UniXcoder SimCSE instead of GraphCodeBERT SimCSE (ablation) | Similarity model · APPS (GPT-3.5-generated) | AUROC: 83.25 -> 86.37 | Helped |
| Increase rewrite count from m=2 to m=4 to m=8 (ablation) | Proposed detector with CodeLlama detector LLM · APPS (Avg.) | AUROC: 73.75 -> 78.49 -> 80.16 | Helped |
| Increase rewrite count from m=2 to m=4 to m=8 (ablation) | Proposed detector with CodeLlama detector LLM · MBPP (Avg.) | AUROC: 74.54 -> 77.99 -> 81.81 | Helped |
Surprising findings
- GPTZero and OpenAI-Detector perform no better than random guessing on code (average AUROC around 46-60).
- Detecting correct code is harder than detecting incorrect code for both the proposed method and baselines.
Limitations
- Performance drops when developers rename identifiers (degrading to random at 50% replacement).
- Benchmarks are limited to 770 APPS and 233 MBPP problems per generator (GPT-4 only 100), and Python-focused with a C++ extension.
- DetectGPT outperforms the method on code longer than 1000 characters.
Experimental setup
Datasets: CodeSearchNet APPS MBPP Code Contest
Benchmarks: APPS synthetic code detection MBPP synthetic code detection Code Contest C++
Metrics: AUROC
Baselines: GPTZero OpenAI-Detector log P(x) LogRank Rank Entropy DetectGPT LRR NPR
Builds on: SimCSE DetectGPT GraphCodeBERT chain-of-thought prompting
Closest papers by meaning
- NextCoder: Robust Adaptation of Code LMs to Diverse Code Edits (ICML 2025 · similarity 0.88)
- Training Language Models on Synthetic Edit Sequences Improves Code Synthesis (ICLR 2025 · similarity 0.87)
- ChatGPT-Generated Code Assignment Detection Using Perplexity of Large Language Models (Student Abstract) (AAAI 2024 · similarity 0.87)
- Detecting AI-Generated Code Assignments Using Perplexity of Large Language Models (AAAI 2024 · similarity 0.87)
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code (ICLR 2025 · similarity 0.87)
- Training Language Models to Generate Quality Code with Program Analysis Feedback (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.