Research map / Large language models / Language, safety and interpretability
Multiscale Positive-Unlabeled Detection of AI-Generated Texts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames AI-generated text detection as a partial positive-unlabeled problem and trains BERT/RoBERTa detectors with a length-sensitive Multiscale PU loss plus sentence-level random deletion augmentation, improving short-text detection.
Problem
Detectors of LLM-generated text fail on short texts (SMS, tweets, reviews), where short machine texts are often indistinguishable from human text, while long-text detection must not be sacrificed.
Approach
Treat short machine texts as partially unlabeled and apply a PU loss whose class prior varies with text length, estimated by an abstract recurrent token-level discriminator; combined with a Text Multiscaling module that randomly deletes sentences to create varied-length training texts.
Key results
- RoBERTa-MPU reaches 85.31 F1 on HC3-English-Sent versus 58.60 for finetuned RoBERTa and 81.46 for RoBERTa-Stylo, while keeping 98.40 on HC3-English-Full.
- On TweepFake, RoBERTa-MPU achieves 91.4 accuracy, above RoBERTa-Stylo (91.1) and RoBERTa-Finetuned (89.6).
- On HC3-Chinese-Sent, RoBERTa-MPU reaches 89.37 F1 versus 83.07 for finetuned RoBERTa.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply MPU training | RoBERTa-Finetuned · TweepFake | accuracy: 89.6 -> 91.4 | Helped |
| apply MPU training | RoBERTa-Finetuned · HC3-En-Sent | F1: 58.60$\pm$10.53 -> 85.31$\pm$1.80 | Helped |
| apply MPU training | RoBERTa-Finetuned · HC3-En-Full | F1: 97.42$\pm$0.92 -> 98.40$\pm$0.31 | Helped |
| apply MPU training | RoBERTa-Finetuned · HC3-Ch-Sent | F1: 83.07$\pm$6.85 -> 89.37$\pm$1.94 | Helped |
| omit short training texts from backward operations (ablation) | RoBERTa finetuning · HC3-En-Sent | F1: 58.60 $\pm$ 10.53 -> 62.42 $\pm$ 5.60 | Helped |
| add Text Multiscaling only (ablation) | RoBERTa finetuning without MPU loss · HC3-English Sent | F1: 58.60$\pm$10.53 -> 82.76$\pm$2.76 | Helped |
| add Text Multiscaling only (ablation) | RoBERTa finetuning without MPU loss · HC3-English Full | F1: 97.42$\pm$0.92 -> 96.42$\pm$2.27 | Hurt |
| add MPU loss only (ablation) | RoBERTa finetuning without Text Multiscaling · HC3-English Sent | F1: 58.60$\pm$10.53 -> 45.30$\pm$8.78 | Hurt |
| add MPU loss only (ablation) | RoBERTa finetuning without Text Multiscaling · HC3-English Full | F1: 97.42$\pm$0.92 -> 97.48$\pm$2.41 | No clear effect |
| replace Multiscale PU loss with ordinary PU loss (ablation) | RoBERTa-MPU · HC3-English Sent | F1: 83.53$\pm$3.14 -> 85.31$\pm$1.80 | Helped |
| replace non-negative PU loss with unbiased PU loss (ablation) | RoBERTa-MPU · HC3-English Full | F1: 97.90$\pm$0.25 -> 98.40$\pm$0.31 | Helped |
| set token prior p to 0.3 instead of 0.2 (ablation) | RoBERTa-MPU · HC3-English Full | F1: 98.40$\pm$0.31 -> 96.81$\pm$1.70 | Hurt |
Surprising findings
- The HC3 English corpus has an artifact: human texts contain extra spaces before punctuation (token 479), so a simple rule reaches 82.12% F1 on the sentence-level test set, beating the reported finetuned RoBERTa-base score (81.89%).
- Adding Text Multiscaling alone lowers full-length F1 (96.42 vs 97.42 on HC3-English), and MPU loss alone hurts short-text F1 (45.30), so the two components must be combined.
Limitations
- Short-text gains rely on sentence-level deletion, which can create label noise for long-text classes and lowers full-text F1 without MPU loss.
- The HC3 benchmark contains a spacing artifact that the authors note may inflate reported results.
- Hyperparameters (p, gamma, p_sent) were tuned on the test benchmarks, and DetectGPT thresholds were selected on the test set, which the authors note gives an upper bound.
Experimental setup
Datasets: TweepFake HC3-English HC3-Chinese
Benchmarks: TweepFake HC3-English full and sentence-level HC3-Chinese full and sentence-level
Metrics: accuracy F1 score
Baselines: GLTR PPL OpenAI Detector DetectGPT BERT-Finetuned RoBERTa-Finetuned RoBERTa-Stylo
Builds on: nnPU loss unbiased PU loss EDA RoBERTa
Closest papers by meaning
- Position: On the Possibilities of AI-Generated Text Detection (ICML 2024 · similarity 0.90)
- Human Texts Are Outliers: Detecting LLM-generated Texts via Out-of-distribution Detection (NeurIPS 2025 · similarity 0.88)
- Training-free LLM-generated Text Detection by Mining Token Probability Sequences (ICLR 2025 · similarity 0.88)
- Detecting Machine-Generated Texts by Multi-Population Aware Optimization for Maximum Mean Discrepancy (ICLR 2024 · similarity 0.87)
- DALD: Improving Logits-based Detector without Logits from Black-box LLMs (NeurIPS 2024 · similarity 0.87)
- DeTeCtive: Detecting AI-generated Text via Multi-Level Contrastive Learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.