Research map / Large language models / Alignment and preferences
ReMoDetect: Reward Models Recognize Aligned LLM's Generations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that aligned LLMs' texts get higher reward-model scores than human text, and proposes ReMoDetect, which continually fine-tunes a reward model with replay and adds human/LLM mixed-text preference training for zero-overhead, single-pass detection of LLM-generated text.
Problem
Detecting text generated by recent powerful aligned LLMs (e.g., GPT-4, Claude 3) is challenging, since existing zero-shot and supervised detectors were mostly built for non-aligned models and generalize poorly across LLMs.
Approach
Aligned LLMs are optimized to maximize human preference, so their texts receive higher reward-model scores than human text; the method further sharpens this separation by continual Bradley-Terry fine-tuning with a replay-buffer regularizer and adds a median-preference class of human/LLM rephrased mixed texts.
Key results
- Average AUROC on the Fast-DetectGPT benchmark rises from 91.9% (best prior, Fast-DetectGPT) to 95.8%; on MGTBench average AUROC is 99.9%.
- Under T5-3B rephrasing attack, the method retains higher AUROC than baselines (e.g., GPT4 Turbo 94.6% attacked vs 79.9-95.2% for baselines' originals), with a relative drop of only -4.4%.
- Component ablation on GPT4 texts: AUROC goes 79.0 (no continual tuning) -> 90.5 -> 95.5 (with replay) -> 97.9 (with mixed-text reward modeling).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReMoDetect instead of the strongest prior benchmark detector | Aligned LLM text detection · Fast-DetectGPT benchmark average | AUROC (%): 91.9 -> 95.8 | Helped |
| Use ReMoDetect instead of the strongest prior benchmark detector | Aligned LLM text detection · MGTBench average | AUROC (%): 94.9 -> 99.9 | Helped |
| Use ReMoDetect instead of GPTZero | Aligned LLM text detection · Fast-DetectGPT benchmark average, GPT4 | AUROC (%): 88.5 -> 97.9 | Helped |
| Add continual preference fine-tuning without replay buffers (ablation) | OpenAssistant reward model · Fast-DetectGPT benchmark, GPT4 | AUROC: 79.0 -> 90.5 | Helped |
| Add replay buffers to continual preference fine-tuning (ablation) | Continually fine-tuned reward model · Fast-DetectGPT benchmark, GPT4 | AUROC: 90.5 -> 95.5 | Helped |
| Add mixed Human/LLM text reward modeling (ablation) | Continually fine-tuned reward model with replay · Fast-DetectGPT benchmark, GPT4 | AUROC: 95.5 -> 97.9 | Helped |
| Train ReMoDetect rather than use the unmodified reward model (ablation) | OpenAssistant reward model · Fast-DetectGPT benchmark, GPT4 | AUROC: 79.0 -> 97.9 | Helped |
| Use ReMoDetect rather than ChatGPT-Detector | Detection on unseen domain and unseen LLM · WritingPrompts-small, Claude3 Opus | AUROC: 24.1 -> 99.5 | Helped |
| Use ReMoDetect rather than Fast-DetectGPT | Detection after T5-3B rephrasing attack · Fast-DetectGPT benchmark average, GPT4 Turbo | AUROC (%): 87.3 -> 94.6 | Helped |
| Use ReMoDetect rather than Fast-DetectGPT with shorter responses | Claude3 Opus text detection · 60 words | AUROC: 91.8 -> 94.1 | Helped |
| Use ReMoDetect rather than Fast-DetectGPT | Detection of non-RLHF aligned and SFT-only text · Fast-DetectGPT benchmark average | AUROC (%): 91.2 -> 95.3 | Helped |
| Use ReMoDetect rather than Fast-DetectGPT | Aligned LLM text detection · Fast-DetectGPT benchmark average | Detection time (secs): 62.7 -> 8.7 | Helped |
Surprising findings
- Aligned LLM texts receive higher reward-model scores than human-written texts even without any detector training, so a zero-shot reward model already detects them (e.g., AUROC 92.8% on GPT-4).
- Ready-made reward model (without training) fails to match the detector on rephrasing attack in some cases; SFT-only OLMo-7B texts are also detected, suggesting SFT alone implicitly captures human preference.
Limitations
- Reward models are less accessible than open LLMs, limiting the approach's breadth.
- Training used only ChatGPT texts from HC3 and mixed texts from Llama3-70B, so the method is trained on a narrow set of generators.
- Evaluation of GPT-4 and Claude texts on rephrasing attack and length robustness relies on the authors' chosen settings; Gemini pro results on XSum are weaker than GPTZero.
Experimental setup
Datasets: HC3 Anthropic/hh-rlhf Dahoas/synthetic-instruct-gptj-pairwise Fast-DetectGPT benchmark (PubMed, XSum, WritingPrompts-small) MGTBench (Essay, Reuters, WritingPrompts)
Benchmarks: Fast-DetectGPT benchmark MGTBench
Metrics: AUROC AUPR TPR at 1% FPR
Baselines: Log-likelihood Rank DetectGPT LRR NPR Fast-DetectGPT OpenAI-Detector ChatGPT-Detector GPTZero Log Rank Entropy GLTR
Builds on: Fast-DetectGPT Bradley-Terry reward modeling ChatGPT-Detector DetectGPT mixup
Closest papers by meaning
- Aligning to Thousands of Preferences via System Message Generalization (NeurIPS 2024 · similarity 0.90)
- Reward-Augmented Data Enhances Direct Preference Alignment of LLMs (ICML 2025 · similarity 0.90)
- GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-Time Alignment (ICLR 2025 · similarity 0.89)
- RMB: Comprehensively benchmarking reward models in LLM alignment (ICLR 2025 · similarity 0.89)
- Larger or Smaller Reward Margins to Select Preferences for LLM Alignment? (ICML 2025 · similarity 0.89)
- MM-RLHF: The Next Step Forward in Multimodal LLM Alignment (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.