Research map / Large language models / Language, safety and interpretability
AI as Humanity’s Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Creativity Index, a web-reconstruction-based originality metric computed with a novel DJ Search dynamic programming algorithm, showing human texts score higher than LLM texts and that the metric works for zero-shot machine text detection.
Problem
Quantifying the linguistic creativity of LLM-generated text versus human-written text at scale, which rubric-based human evaluation cannot do.
Approach
Measure creativity as the L-uniqueness-based fraction of a text's n-grams (verbatim or WMD-based semantic matches) not found in a web reference corpus, integrated over n-gram length; DJ Search is a two-pointer dynamic program that needs only linear many corpus lookups, using Infini-gram for exact matches and WMD on BM25-retrieved documents for near-verbatim matches.
Key results
- Human texts have a Creativity Index on average 66.2% higher than LLMs, with 52.2% higher in novel writing on verbatim matches.
- RLHF reduces LLM Creativity Index by an average of 30.1% (verbatim matches).
- For zero-shot machine text detection, the method surpasses DetectGPT by 30.2% AUROC and outperforms supervised GhostBuster in five of six domains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use human-written text rather than LLM-generated text | Human and LLM writing · Novel writing | Creativity Index: 52.2% | Helped |
| Use human-written text rather than LLM-generated text | Human and LLM writing · Poetry composition | Creativity Index: 31.1% | Helped |
| Use human-written text rather than LLM-generated text | Human and LLM writing · Speech drafting | Creativity Index: 115.3% | Helped |
| Add semantic matching to verbatim matching (ablation) | Creativity Index · Novel writing | Human–LLM Creativity Index gap: 52.2% -> 102.5% | Helped |
| Encourage creativity through prompt wording (ablation) | ChatGPT · Novel writing | Creativity Index | No clear effect |
| Increase top-p decoding value (ablation) | ChatGPT · Novel writing | Creativity Index | No clear effect |
| Increase the number of human-text sentences in the prompt (ablation) | ChatGPT · Novel writing | Creativity Index | No clear effect |
| Vary model size (ablation) | LLaMA 2 Chat and Tulu 2 · Novel writing | Creativity Index | No clear effect |
| Apply RLHF alignment | GPT, LLaMA 2, and OLMo · Novel writing, verbatim matching | Creativity Index: 30.1% | Hurt |
| Apply RLHF alignment | GPT, LLaMA 2, and OLMo · Novel writing, verbatim and semantic matching | Creativity Index: 8.9% | Hurt |
| Use the top 50 matching reference documents | Machine and human texts · Novel writing, L=5 | L-uniqueness: 32.8%; 56.6%; 73.4% lower | Mixed |
| Use Creativity Index instead of DetectGPT for zero-shot detection | Machine-text detector · Multiple detection domains | AUROC: 30.2% | Helped |
Surprising findings
- Prompting models with explicit creativity instructions made little statistically significant difference to the Creativity Index.
- Newer GPT-4 appears less inflated when measured against a model-generated reference corpus, yet humans still score 30.3% higher.
Limitations
- Creativity Index depends on the chosen reference corpus (RedPajama cutoff), which may inflate scores for closed models trained on private data.
- Overlap with web text does not conclusively indicate memorization of a specific document.
- Human authors studied are high-quality published writers, not representative of average human writing.
Experimental setup
Datasets: RedPajama BookMIA PoemHunter poems American Rhetoric speeches Misinfo Reaction Frames XSum NaturalProofs Goodreads book lists
Benchmarks: BookMIA PoemHunter poems American Rhetoric speeches XSum fake news detection NaturalProofs theorem proofs
Metrics: Creativity Index L-uniqueness AUROC Mann-Whitney U p-value
Baselines: DetectGPT OpenAI RoBERTa detector GhostBuster
Builds on: Infini-gram Word Mover's Distance DetectGPT BM25
Closest papers by meaning
- Quality-Diversity through AI Feedback (ICLR 2024 · similarity 0.87)
- Predicting Empirical AI Research Outcomes with Language Models (NeurIPS 2025 · similarity 0.87)
- Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond) (NeurIPS 2025 · similarity 0.86)
- CLAWS:Creativity detection for LLM-generated solutions using Attention Window of Sections (NeurIPS 2025 · similarity 0.86)
- CreBench: Human-Aligned Creativity Evaluation from Idea to Process to Product (AAAI 2026 · similarity 0.86)
- Raidar: geneRative AI Detection viA Rewriting (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.