Research map / Large language models / Language, safety and interpretability
DNA-DetectLLM: Unveiling AI-Generated Text via a DNA-Inspired Mutation-Repair Paradigm
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DNA-DetectLLM, a zero-shot AI-generated text detector that scores how much token-level "repair" an input needs to match its greedy-decoded ideal sequence, and reports SOTA AUROC/F1 on multiple benchmarks.
Problem
Detecting AI-generated text is increasingly hard because feature distributions of human and AI text overlap; existing training-free detectors rely on fixed statistical scores that blur under this overlap.
Approach
Builds an ideal AI sequence by greedy decoding under a reference LM, then measures the cumulative effort (conditional perplexity-based score) of repairing non-argmax tokens to reach it; a simplified repair score averages the input score and the ideal-sequence conditional score, derived as the expectation over random repair orders.
Key results
- Average AUROC of 98.30% on XSum/WritingPrompts/Arxiv self-generated data, a 0.93% relative gain over the best baseline.
- Average AUROC 90.86% and F1 87.02% across M4, DetectRL, and RealDet, with claimed relative AUROC gain of 5.55% over baselines.
- Under 1% token edits and DIPPER paraphrasing on GPT-4 Turbo text, it attains AUROC 93.37 (insertion) and 98.02 (paraphrase), the top scores in those tables.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DNA-DetectLLM instead of the strongest baseline, Binoculars | AI-generated text detection · M4, DetectRL Multi-LLM, DetectRL Multi-Domain, RealDet (average) | AUROC: 5.55% | Helped |
| use DNA-DetectLLM instead of the strongest baseline, Binoculars | AI-generated text detection · M4, DetectRL Multi-LLM, DetectRL Multi-Domain, RealDet (average) | F1: 2.08% | Helped |
| use DNA-DetectLLM instead of the strongest baseline | AI-generated text detection · DetectRL Multi-LLM | AUROC: 6.92% | Helped |
| use DNA-DetectLLM instead of the strongest baseline | AI-generated text detection · DetectRL Multi-Domain | AUROC: 13.92% | Helped |
| change repair order to low-to-high (ablation) | DNA-DetectLLM · XSum, WritingPrompt, Arxiv (average) | AUROC: 98.30 -> 97.43 | Hurt |
| change repair order to high-to-low (ablation) | DNA-DetectLLM · XSum, WritingPrompt, Arxiv (average) | AUROC: 98.30 -> 98.16 | Hurt |
| repair tokens in input order (ablation) | DNA-DetectLLM · XSum, WritingPrompt, Arxiv (average) | AUROC: 98.30 -> 98.23 | Hurt |
| use the simplified repair score rather than recalculating conditional scores after each token repair (ablation) | DNA-DetectLLM · XSum, WritingPrompt, Arxiv, M4, RealDet | time cost: 14.11, 14.45, 14.55 -> 0.78 | Helped |
| use the Llama-2-7B + Llama-7B model pair instead of Falcon-7B-Instruct + Falcon-7B (ablation) | DNA-DetectLLM · public benchmark datasets | AUROC: 92.4% and 90.7% | Helped |
Surprising findings
- Using Llama-2-7B + Llama-7B as the scoring pair slightly outperforms the default Falcon pair (AUROC 92.4% vs 90.7%).
Limitations
- Efficiency measured only with small batch size due to memory constraints, so large-scale real-time behavior is not explored.
- Relative improvements reported over baselines are sensitive to the choice of reference and observer models.
- No error bars or statistical significance reported.
Experimental setup
Datasets: XSum WritingPrompts Arxiv M4 DetectRL RealDet HC3 PubMedQA
Benchmarks: XSum WritingPrompts Arxiv M4 DetectRL Multi-LLM DetectRL Multi-Domain RealDet PubMedQA
Metrics: AUROC F1 score
Baselines: OpenAI-D Biscope R-Detect Entropy Likelihood LogRank DetectGPT Fast-DetectGPT Binoculars Lastde++ GPTZero ImBD
Builds on: Fast-DetectGPT Binoculars DetectGPT DNA-GPT Lastde++
Closest papers by meaning
- DNA-GPT: Divergent N-Gram Analysis for Training-Free Detection of GPT-Generated Text (ICLR 2024 · similarity 0.90)
- Position: On the Possibilities of AI-Generated Text Detection (ICML 2024 · similarity 0.89)
- Raidar: geneRative AI Detection viA Rewriting (ICLR 2024 · similarity 0.87)
- DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios (NeurIPS 2024 · similarity 0.87)
- DETree: DEtecting Human-AI Collaborative Texts via Tree-Structured Hierarchical Representation Learning (NeurIPS 2025 · similarity 0.86)
- Few-Shot Detection of Machine-Generated Text using Style Representations (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.