Research map / Label-efficient and robust learning / OOD and anomaly detection
Advancing Machine-Generated Text Detection from an Easy to Hard Supervision Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an easy-to-hard supervision framework for machine-generated text detection in which a simple supervisor trained on concatenated longer texts (gated by the detector) jointly trains with and indirectly optimizes the target detector, evaluated on DetectRL and Essay.
Problem
MGT detection training assumes hard labels are golden, but boundary ambiguity (human-machine collaboration, human-like MGT) and detector superintelligence make supervision inexact and hard to evaluate.
Approach
Train an easy supervisor on longer spliced texts, with the detector embedded as a gating mechanism (Gumbel-softmax) so the supervisor's performance acts as a lower bound on the detector; joint optimization indirectly improves the detector toward golden labels, unlike knowledge distillation.
Key results
- RADAR-E raises DetectRL sentence-level average TPR@FPR-1% from 27.98% to 31.65%.
- On Essay sentence-level, RADAR average TPR@FPR-1% rises from 26.19% to 28.98% with RADAR-E.
- On Essay paragraph-level, RADAR-E averages 74.78% TPR@FPR-1% versus 69.51% for RADAR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply easy-to-hard supervision to RADAR | RADAR · DetectRL sentence-level, cross-LLM average | TPR@FPR-1%: 27.98 -> 31.65 | Helped |
| Apply easy-to-hard supervision to RADAR | RADAR · DetectRL paragraph-level, cross-LLM average | TPR@FPR-1%: 72.35 -> 74.27 | Helped |
| Apply easy-to-hard supervision to RADAR | RADAR · Essay sentence-level, cross-LLM average | TPR@FPR-1%: 26.19 -> 28.98 | Helped |
| Apply easy-to-hard supervision to RADAR | RADAR · Essay paragraph-level, cross-LLM average | TPR@FPR-1%: 69.51 -> 74.78 | Helped |
| Apply easy-to-hard supervision to MPU | MPU · DetectRL sentence-level, cross-LLM average | TPR@FPR-1%: 22.89 -> 24.85 | Helped |
| Apply easy-to-hard supervision to MPU | MPU · DetectRL paragraph-level, cross-LLM average | TPR@FPR-1%: 63.92 -> 67.52 | Helped |
| Apply easy-to-hard supervision to MPU | MPU · Essay sentence-level, cross-LLM average | TPR@FPR-1%: 14.35 -> 16.58 | Helped |
| Apply easy-to-hard supervision to MPU | MPU · Essay paragraph-level, cross-LLM average | TPR@FPR-1%: 53.80 -> 57.11 | Helped |
| Apply easy-to-hard supervision to ChatGPT-D | ChatGPT-D · DetectRL sentence-level, cross-LLM average | TPR@FPR-1%: 8.99 -> 10.48 | Helped |
| Apply easy-to-hard supervision to ChatGPT-D | ChatGPT-D · DetectRL paragraph-level, cross-LLM average | TPR@FPR-1%: 22.81 -> 25.83 | Helped |
| Apply easy-to-hard supervision to ChatGPT-D | ChatGPT-D · Essay sentence-level, cross-LLM average | TPR@FPR-1%: 11.33 -> 11.94 | Helped |
| Apply easy-to-hard supervision to ChatGPT-D | ChatGPT-D · Essay paragraph-level, cross-LLM average | TPR@FPR-1%: 41.40 -> 42.06 | Helped |
Surprising findings
- Knowledge distillation from a strong teacher (RADAR) often decreased student performance in most settings.
- Cross-LLM detection is not always inferior to within-LLM detection; RADAR did worse on PaLM than ChatGPT texts.
Limitations
- Only applicable to model-based detectors with relatively good performance; metric-based methods not addressed.
- Golden label approximated as proportion of pure machine content, which is imperfect.
- Splicing-based longer-text construction is a simplification of real mixed text.
Experimental setup
Datasets: Essay DetectRL
Benchmarks: DetectRL Essay
Metrics: TPR@FPR-1% AUROC
Baselines: Likelihood Log-Rank Entropy NPR DetectGPT Fast-DetectGPT ChatGPT-D MPU RADAR Knowledge Distillation
Builds on: knowledge distillation label smoothing Gumbel softmax Fast-DetectGPT
Closest papers by meaning
- Position: On the Possibilities of AI-Generated Text Detection (ICML 2024 · similarity 0.87)
- DALD: Improving Logits-based Detector without Logits from Black-box LLMs (NeurIPS 2024 · similarity 0.86)
- Log-Likelihood, Simpson’s Paradox, and the Detection of Machine-Generated Text (NeurIPS 2026 · similarity 0.85)
- Multiscale Positive-Unlabeled Detection of AI-Generated Texts (ICLR 2024 · similarity 0.85)
- DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios (NeurIPS 2024 · similarity 0.85)
- Detecting Machine-Generated Texts by Multi-Population Aware Optimization for Maximum Mean Discrepancy (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.