Research map / Large language models / Language, safety and interpretability
DPIC: Decoupling Prompt and Intrinsic Characteristics for LLM Generated Text Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DPIC, a black-box LLM-generated text detector that reconstructs a candidate text's prompt with an auxiliary LLM, regenerates text from that prompt, and classifies with a Siamese-embedding classifier on the candidate and regenerated text.
Problem
Detecting texts generated by black-box LLMs (e.g., GPT-4, Claude3) with high quality and cross-domain generalization, where model logits are unavailable and surrogate-model zero-shot detectors degrade.
Approach
Decouple the prompt from intrinsic generator characteristics: reconstruct the prompt from the candidate text with an auxiliary LLM, regenerate text from that prompt, and train a classifier on embeddings of the candidate and regenerated texts so the detector focuses on intrinsic characteristics rather than topic.
Key results
- With ChatGPT as auxiliary LLM, DPIC averages 96.34%, 97.34%, and 98.78% AUROC on ChatGPT, GPT-4, and Claude3 texts, versus 96.15%, 90.58%, 95.87% for the best open-source baseline Fast-DetectGPT.
- Under DIPPER paraphrase attack on XSum ChatGPT texts, DPIC (ChatGPT) reaches 100% AUROC; under back-translation it reaches 99.72%.
- Concatenation of candidate and regenerated embeddings yields 98.21% AUROC on Writing/ChatGPT versus 89.81% for absolute difference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DPIC with ChatGPT as auxiliary model instead of Fast-DetectGPT | LLM-generated text detection · GPT-4-generated texts, XSum/Writing/PubMed average | AUROC: 0.9058 -> 0.9734 | Helped |
| Use DPIC with ChatGPT as auxiliary model instead of Fast-DetectGPT | LLM-generated text detection · Claude3-generated texts, XSum/Writing/PubMed average | AUROC: 0.9587 -> 0.9878 | Helped |
| Use DPIC with ChatGPT as auxiliary model instead of Fast-DetectGPT | LLM-generated text detection · ChatGPT-generated texts, XSum/Writing/PubMed average | AUROC: 0.9615 -> 0.9634 | Helped |
| Use concatenation instead of absolute difference or dot product to combine feature vectors (ablation) | DPIC · Writing dataset generated by ChatGPT | AUROC: 89.81% and 93.84% ... 98.21% | Helped |
| Use supervised training for DNA-GPT rather than its zero-shot detector (ablation) | DNA-GPT · Datasets generated by ChatGPT, GPT4, and Claude3 | AUROC: 64.42%, 55.29%, and 68.49% -> 88.39%, 85.50%, and 91.51% | Helped |
| Add reconstructed-prompt semantic guidance to DNA-GPT (supervised) (ablation) | DNA-GPT (supervised) · PubMed dataset generated by ChatGPT, GPT4, and Claude3 | detection accuracy: 74.93%, 76.66%, and 79.52% -> 79.28%, 82.18%, and 86.46% | Helped |
| Use prompt-guided regeneration instead of truncated completion (ablation) | DNA-GPT (prompt) · XSum, Writing, and PubMed datasets | AUROC | Helped |
Surprising findings
- Vicuna-7b as a cheap open-source auxiliary model achieves detection performance close to ChatGPT as auxiliary (e.g., 95.58% vs 96.34% average on ChatGPT texts).
Limitations
- Prompt reconstruction is imperfect; full prompt recovery is acknowledged as almost impossible, so reconstructed prompts can differ from originals.
- Relies on a supervised training set (HC3) whose generators (ChatGPT or Vicuna) differ from the test generators.
- Evaluation uses a limited set of closed-source generators and domains; the paper does not report inference cost of the regeneration pipeline.
Experimental setup
Datasets: HC3 XSum WritingPrompts PubMedQA Wiki (constructed) CovidQA SUMMAC
Benchmarks: XSum WritingPrompts PubMedQA Wiki CovidQA SUMMAC
Metrics: AUROC
Baselines: RoBERTa-base RoBERTa-large RADAR Likelihood Entropy LogRank LRR DNA-GPT DetectGPT NPR Fast-DetectGPT
Builds on: DNA-GPT Fast-DetectGPT RADAR DetectGPT
Closest papers by meaning
- DALD: Improving Logits-based Detector without Logits from Black-box LLMs (NeurIPS 2024 · similarity 0.90)
- DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios (NeurIPS 2024 · similarity 0.90)
- Training-free LLM-generated Text Detection by Mining Token Probability Sequences (ICLR 2025 · similarity 0.89)
- Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detection (ICLR 2025 · similarity 0.88)
- Zero-Shot Detection of LLM-Generated Text via Implicit Reward Model (NeurIPS 2025 · similarity 0.88)
- OUTFOX: LLM-Generated Essay Detection Through In-Context Learning with Adversarially Generated Examples (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.