Research map / Large language models / Language, safety and interpretability
Imitate Before Detect: Aligning Machine Stylistic Preference for Machine-Revised Text Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Imitate Before Detect (ImBD), which aligns a scoring LM to machine-revision style via DPO-style style preference optimization on 500 human/GPT-3.5-polished pairs, then detects machine-revised text with Style-CPC.
Problem
Detecting machine-revised text (rewrite, expand, polish) of human-written content is hard because existing logit-based detectors are misled by human-contributed content, causing large AUROC drops versus pure machine-generated text.
Approach
Fine-tune a scoring LM with a DPO-style preference objective on content-matched (machine-revised, human) pairs so its token distribution imitates machine style, then compute style-conditional probability curvature (Style-CPC) with this aligned model to separate machine-revised from human text.
Key results
- On GPT-3.5 polished text, ImBD averages 0.9449 AUROC vs 0.7933 for Fast-DetectGPT (about 15% gain on polish task).
- On open-source-model polished text, ImBD averages 0.9550 AUROC vs 0.8261 for Fast-DetectGPT.
- ImBD (0.9351 GPTZero comparison: 0.9449 avg) surpasses GPTZero (0.9351 avg) on GPT-3.5 polished text with only 1,000 training samples and about 220 s training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add machine-style imitation with SPO (ablation) | Fast-DetectGPT detector · GPT-3.5 polished text | AUROC: 0.79 -> 0.95 | Helped |
| add machine-style imitation with SPO (ablation) | Fast-DetectGPT detector · GPT-4o polished text | AUROC: 0.69 -> 0.89 | Helped |
| replace SFT with SPO (ablation) | ImBD · GPT-3.5 polished text | AUROC: 0.65 -> 0.95 | Helped |
| replace SFT with SPO (ablation) | ImBD · GPT-4o polished text | AUROC: 0.67 -> 0.89 | Helped |
| replace SFT trained on 3x samples with SPO (ablation) | ImBD · GPT-3.5 polished text | AUROC: 0.65 -> 0.95 | Helped |
| replace SFT trained on 3x samples with SPO (ablation) | ImBD · GPT-4o polished text | AUROC: 0.65 -> 0.89 | Helped |
| replace RLHF with SPO (ablation) | ImBD · GPT-3.5 polished text | AUROC: 0.80 -> 0.95 | Helped |
| replace RLHF with SPO (ablation) | ImBD · GPT-4o polished text | AUROC: 0.66 -> 0.89 | Helped |
| replace ORPO with SPO (ablation) | ImBD · GPT-3.5 polished text | AUROC: 0.86 -> 0.95 | Helped |
| replace ORPO with SPO (ablation) | ImBD · GPT-4o polished text | AUROC: 0.71 -> 0.89 | Helped |
| use ImBD instead of Fast-DetectGPT | GPT-3.5 polished-text detection · GPT-3.5 polished text, average across domains | AUROC: 0.7933 -> 0.9449 | Helped |
| use ImBD instead of Fast-DetectGPT | GPT-4o polished-text detection · GPT-4o polished text, average across domains | AUROC: 0.6931 -> 0.8899 | Helped |
Surprising findings
- Supervised RoBERTa detectors perform near chance on machine-revised text, and several logit baselines fall below 0.5 AUROC on some settings.
- The SFT variant trained on 3x data underperforms SPO, suggesting preference optimization matters more than data volume.
Limitations
- Training data covers only GPT-3.5-generated revisions; generalization depends on the revision model.
- Multi-sample perturbation baselines are much costlier, but ImBD relies on a sampling step and a fixed threshold epsilon.
Experimental setup
Datasets: XSum SQuAD WritingPrompts PubMedQA
Benchmarks: XSum SQuAD WritingPrompts PubMedQA
Metrics: AUROC
Baselines: RoBERTa-base RoBERTa-large GPTZero Likelihood Entropy LogRank LRR NPR DNA-GPT DetectGPT Fast-DetectGPT RLHF
Builds on: DPO Fast-DetectGPT DetectGPT
Closest papers by meaning
- Few-Shot Detection of Machine-Generated Text using Style Representations (ICLR 2024 · similarity 0.90)
- Alignment Imprint: Zero-Shot AI-Generated Text Detection via Provable Preference Discrepancy (NeurIPS 2026 · similarity 0.88)
- Detecting Machine-Generated Texts by Multi-Population Aware Optimization for Maximum Mean Discrepancy (ICLR 2024 · similarity 0.88)
- DALD: Improving Logits-based Detector without Logits from Black-box LLMs (NeurIPS 2024 · similarity 0.87)
- ReMoDetect: Reward Models Recognize Aligned LLM's Generations (NeurIPS 2024 · similarity 0.87)
- DeTeCtive: Detecting AI-generated Text via Multi-Level Contrastive Learning (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.