Research map / Large language models / Language, safety and interpretability
DALD: Improving Logits-based Detector without Logits from Black-box LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes a LoRA surrogate LLM on a small (about 5K) set of target-model outputs so logits-based zero-shot detectors (Fast-DetectGPT, DNA-GPT, DetectGPT) work on black-box LLM text without target logits.
Problem
Logits-based zero-shot detectors need a surrogate model whose distribution matches the black-box target LLM; mismatch and frequent closed-source model updates degrade detection accuracy.
Approach
Align the surrogate model's distribution with the target black-box LLM by self-supervised LoRA fine-tuning on a small corpus of publicly shared outputs (prompt-masked cross-entropy), then plug the aligned surrogate into existing logits-based detectors; a theorem bounds the curvature gap between surrogate and target after alignment.
Key results
- On GPT-4-0613 PubMed, DALD with Llama2-7B surrogate reaches AUROC 0.9785 versus 0.8179 for Fast-DetectGPT.
- On Claude-3 PubMed, Fast-DetectGPT with aligned surrogate improves from 0.8900 to 0.9630 AUROC; DALD on Claude-3 Writing reaches 0.9981.
- One-for-all surrogate trained only on GPT-4-0613 data achieves about 0.98 or higher AUROC on several other GPT-3.5/GPT-4 versions in PubMed/XSum/Writing.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Align the surrogate with GPT-4 outputs using DALD (ablation) | Fast-DetectGPT · PubMed-GPT-4 | AUROC: 0.8179 -> 0.9785 | Helped |
| Add DALD surrogate fine-tuning (ablation) | Detect-GPT · PubMed-GPT-4 | AUROC: 0.5291 -> 0.7034 | Helped |
| Add DALD surrogate fine-tuning (ablation) | DNA-GPT · PubMed-GPT-4 | AUROC: 0.8947 -> 0.9879 | Helped |
| Add DALD surrogate fine-tuning (ablation) | Fast-DetectGPT · PubMed-GPT-4 | AUROC: 0.8179 -> 0.9785 | Helped |
| Use a single GPT-4-trained DALD surrogate across target models instead of training separately (ablation) | DALD · PubMed-ChatGPT | AUROC: 0.9853 -> 0.9829 | Hurt |
| Train one DALD surrogate on two sources rather than one (ablation) | DALD · XSum-GPT-4 | AUROC: 0.9954 -> 0.9981 | Helped |
| Train one DALD surrogate on three sources rather than two (ablation) | DALD · XSum-Claude-3 | AUROC: 0.9994 -> 0.9996 | Helped |
| Fine-tune Llama2-7B as the surrogate with DALD (ablation) | Fast-DetectGPT · PubMed-Claude-3 | AUROC: 0.8876 -> 0.9424 | Helped |
| Fine-tune Llama3-8B as the surrogate with DALD (ablation) | Fast-DetectGPT · PubMed-Claude-3 | AUROC: 0.7764 -> 0.9102 | Helped |
| Fine-tune GPT-Neo-2.7B as the surrogate with DALD (ablation) | Fast-DetectGPT · PubMed-Claude-3 | AUROC: 0.8900 -> 0.8997 | Helped |
| Train on more data sources, from one to three (ablation) | DALD · PubMed-ChatGPT | AUROC: 0.9829 -> 0.9827 | Hurt |
| Increase the test sample count from 150 to 300 (ablation) | DALD · PubMed-ChatGPT | AUROC: 0.9853 -> 0.9842 | Hurt |
Surprising findings
- A surrogate trained on data from a single source model (GPT-4-0613) transfers to other unseen GPT-3.5 and GPT-4 versions with little loss, suggesting closed-source models have similar distributions.
- Detection on the GPT-3.5 ChatGPT PubMed setting shows slight degradation when adding more data sources, while other settings improve.
Limitations
- Evaluated only in English and German; multilingual performance unexplored.
- Requires collecting target-model outputs for the exact model version, so alignment must be redone for new model versions.
- No error bars or statistical significance reported.
Experimental setup
Datasets: WildChat Claude-3-Opus-Instruct-15K XSum WritingPrompts WMT-2016 PubMedQA APPS RAID
Benchmarks: XSum WritingPrompts WMT-2016 PubMedQA APPS RAID
Metrics: AUROC AUPR
Baselines: RoBERTa-base RoBERTa-large Likelihood Entropy LogRank LRR NPR Detect-GPT DNA-GPT Fast-DetectGPT
Builds on: Fast-DetectGPT DetectGPT DNA-GPT LoRA
Closest papers by meaning
- Glimpse: Enabling White-Box Methods to Use Proprietary Models for Zero-Shot LLM-Generated Text Detection (ICLR 2025 · similarity 0.92)
- Training-free LLM-generated Text Detection by Mining Token Probability Sequences (ICLR 2025 · similarity 0.91)
- DPIC: Decoupling Prompt and Intrinsic Characteristics for LLM Generated Text Detection (NeurIPS 2024 · similarity 0.90)
- Human Texts Are Outliers: Detecting LLM-generated Texts via Out-of-distribution Detection (NeurIPS 2025 · similarity 0.90)
- DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios (NeurIPS 2024 · similarity 0.89)
- Log-Likelihood, Simpson’s Paradox, and the Detection of Machine-Generated Text (NeurIPS 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.