Research map / Large language models / Alignment and preferences
POEMetric: The Last Stanza of Humanity
Bingru Li, Han Wang, Hazel Wilkinson
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use human poets instead of best LLM (DeepSeek-R1) | POEMetric · overall poem quality | score (1-5): 3.20 -> 4.22 | Improved |
| use DeepSeek-R1 instead of humans for lexical diversity | POEMetric · lexical diversity | score (1-5): 3.82 -> 3.85 | Improved |
| choose Gemini-2.5-Pro as judge instead of GPT-4o (ablation) | LLM-as-a-judge · agreement with human experts | PAo: 0.438 -> 0.662 | Improved |
| choose Gemini-2.5-Pro as judge instead of DeepSeek-R1 (ablation) | LLM-as-a-judge · discrimination of overall quality | std. dev.: 0.20 -> 0.63 | Improved |
| use thinking models instead of non-thinking | LLMs · poem generation overall score | mean score: GPT-4o and GPT-4 ranked higher than o1 and o3-mini | No clear effect |
| increase parameter count within a model family | LLMs · poem generation overall score | mean score: qualitative | Improved |
| distill DeepSeek-R1 into smaller models | DeepSeek-R1-Distill · poem generation overall score | mean score: qualitative; Distill-Llama-3.3-70B exception | Hurt |
| compare Gemini-2.5-Pro and Llama-3.3-70B-Instruct | LLMs · form accuracy | score (1-5): 2.29 -> 4.26 | Improved |
Closest papers by meaning
- PoetryDiffusion: Towards Joint Semantic and Metrical Manipulation in Poetry Generation (AAAI 2024 · similarity 0.86)
- AI as Humanity’s Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text (ICLR 2025 · similarity 0.85)
- Metaphor Is Not All Attention Needs (NeurIPS 2026 · similarity 0.84)
- Understanding Annotator Perception: Modeling Psychological Inference from First- and Third-Person Annotations (Student Abstract) (AAAI 2025 · similarity 0.84)
- MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models (NeurIPS 2025 · similarity 0.83)
- Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
