Research map / Large language models / Language, safety and interpretability
Death of the Novel(ty): Beyond N-Gram Novelty as a Metric for Textual Creativity
Arkadiy Saakyan, Najoung Kim, Smaranda Muresan, Tuhin Chakrabarty
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| higher n-gram novelty (log-standardized perplexity) of an expression | expression-level creativity judgments · Expert close reading (OLMo, OLMo-2, human) | odds of judged creative: OR 1.96 per SD | Improved |
| use n-gram novelty alone as a creativity filter (top quartile) | n-gram novelty metric · Expert close reading | share judged creative: approx 91% of top-quartile novel expressions not judged creative | Hurt |
| higher n-gram novelty in OLMo-2 generated text | OLMo-2 · Expert close reading | pragmaticality: beta -0.48 | Hurt |
| higher n-gram novelty in OLMo generated text | OLMo · Expert close reading | pragmaticality: beta -0.17 | Hurt |
| higher n-gram novelty in human-written text | human text · Expert close reading | pragmaticality: beta 0.01 | No clear effect |
| switch from human to LLM authorship (GPT-5, Claude 4.1, OLMo, OLMo-2) | passage generation source · Expert close reading | odds of expression judged creative: OR 0.511 (GPT-5), 0.521 (Claude-4.1) vs human | Hurt |
| higher passage-level AI-detector likelihood | passage · Expert close reading | creativity: beta 0.05, p = 0.624 | No clear effect |
| higher writing quality reward model score | passage · Expert close reading | creativity: OR 1.30 | Improved |
| few-shot prompting instead of zero-shot (ablation) | LLM-as-a-Judge close reading · Novel / non-pragmatic expression identification | F1 | No clear effect |
| finetune smaller models on 60% of dataset | OLMo2 7B, Qwen3 8B, Llama-3.1 8B · Close reading task | F1 | Hurt |
| use LLM-J novelty score differences to predict expert preference | GPT-5 few-shot · Style Mimic expert preference | preference coefficient: beta 0.63, OR 1.88 | Improved |
| use LLM-J pragmaticality score differences to predict expert preference | GPT-5 few-shot · Style Mimic expert preference | preference coefficient: beta -0.05, p = 0.832 | No clear effect |
Closest papers by meaning
- AI as Humanity’s Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text (ICLR 2025 · similarity 0.89)
- Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks (NeurIPS 2026 · similarity 0.87)
- Quality-Diversity through AI Feedback (ICLR 2024 · similarity 0.85)
- Metaphor Is Not All Attention Needs (NeurIPS 2026 · similarity 0.85)
- IDEAFix: Evaluation Framework for Creative Defixation Prompting in LLMs (NeurIPS 2026 · similarity 0.84)
- A Tale of Two Structures: Do LLMs Capture the Fractal Complexity of Language? (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
