Research map / Large language models / AI and society
Position: Societal Impacts Research Requires Benchmarks for Creative Composition Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that societal-impacts research needs usage-grounded benchmarks for "creative composition tasks," supported by a thematic analysis of about 2 million prompts from WildChat-1M and LMSYS-Chat-1M.
Problem
Existing AI benchmarks poorly cover everyday creative generation tasks that users actually perform with language models, leaving their societal impacts (homogenization, misinformation, unfair allocation of opportunities) unmeasured.
Approach
Defines "creative composition tasks" (little-c, everyday creativity generation under constraints) and uses bottom-up thematic analysis of real user prompts to show a mismatch between usage categories and existing creative benchmarks, then argues for usage-based benchmarks with value, novelty, and harm dimensions.
Key results
- Roughly 289k (later 154k LMSYS and 135k WildChat) prompts from 43 creative-leaning topic clusters were retained for analysis after filtering from about 2 million prompts.
- Thematic analysis yields 10 main themes covering many tasks, such as title/name generation, business ideation, academic writing, and personal application materials, that have no existing evaluation.
- Prior usage studies estimate creative composition tasks make up roughly 28% to 62% of prompts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use broad task categories to describe creative composition usage | Prior prompt-pattern analyses · LLM user prompts | task coverage detail: 28.2% | Hurt |
| Develop benchmarks grounded in real-world creative composition usage patterns | Societal impacts evaluation of generative AI · Creative composition tasks | benchmark coverage | Helped |
| Rely on existing static benchmarks and leaderboards | LLM evaluation · Creative composition tasks | usage coverage | Hurt |
| Use arena win rates as the sole evaluation for creative composition | Chatbot Arena-style evaluation · Creative composition tasks | ranking representativeness: 96.7% ... 53.5% | Hurt |
| Evaluate creativity on a single response rather than comparing a cohort of outputs | Creative composition evaluation · Novelty | novelty measurement | Hurt |
| Use reference-answer similarity metrics for creative outputs | Creative composition evaluation · Creative generation tasks | novelty and value assessment | Hurt |
| Use cohort-based novelty evaluation | Creative composition evaluation · Monoculture harms | novelty measurement | Helped |
| Combine quality measures with societal-impact measures | Creative composition benchmarks · Creative composition tasks | holistic impact coverage | Helped |
Surprising findings
- Usage is long-tailed: single power users can dominate clusters, e.g., one user's fan-fiction prompts exceed all prompts mentioning resumes.
- Sexual story-writing prompts persist in both datasets even after toxic and adult content filtering, with around 15-25k prompts.
Limitations
- Analysis restricted to English prompts and to two public datasets collected in 2023-2024.
- Thematic analysis relies on manual coding of samples and keyword-based creativity filtering, which may miss or misclassify prompts.
- Position paper offers no implemented benchmark or empirical evaluation of model harms.
Experimental setup
Datasets: WildChat-1M LMSYS-Chat-1M
Metrics: prevalence of creative composition tasks in prompt clusters thematic coverage gap versus existing benchmarks
Baselines: Chatbot Arena GLUE BIG-bench HELM
Builds on: HELM BIG-bench Chatbot Arena
Closest papers by meaning
- Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks (NeurIPS 2026 · similarity 0.84)
- CreBench: Human-Aligned Creativity Evaluation from Idea to Process to Product (AAAI 2026 · similarity 0.84)
- IDEAFix: Evaluation Framework for Creative Defixation Prompting in LLMs (NeurIPS 2026 · similarity 0.84)
- AI as Humanity’s Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text (ICLR 2025 · similarity 0.84)
- AAAR-1.0: Assessing AI’s Potential to Assist Research (ICML 2025 · similarity 0.83)
- Beyond Accuracy: A Cognitive Load Framework for Mapping the Capability Boundaries of Tool-use Agents (AAAI 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.