Research map / Large language models / Language, safety and interpretability

Within-Model vs Between-Prompt Variability in Large Language Models for Creative Tasks

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Uncertainty quantification for LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Variance decomposition of LLM creative outputs on the Alternative Uses Task across 12 LLMs and 10 prompt strategies (100 samples each), finding prompts explain about 36% of originality variance but only about 4% of fluency variance, while model choice and within-model sampling variance dominate fluency.

Problem

How variance in LLM outputs is partitioned between prompt, model choice, and within-model sampling stochasticity, which single-sample evaluations typically conflate.

Approach

Treats sampling as a random factor by generating 100 repeated samples per model-prompt cell and fitting a linear mixed-effects model to estimate ICC-based variance shares for originality and fluency, rather than relying on single generations per prompt.

Key results

• For originality, prompt explains 36.43% of variance, model 40.94%, within-LLM 10.56%, and model x prompt interaction 12.07%.
• For fluency, model explains 51.25% and within-LLM 33.70% of variance, while prompt explains only 4.22%.
• Persona prompt (P6) produced the highest between-model variance (0.112 vs 0.037 for baseline), with Gemma 3 27B showing d=2.11 under persona.

Surprising findings

• Format constraint prompt (P8) reduced idea uniqueness by 5.9 percentage points and fluency, with over 100 exact duplicates, as a byproduct of the no-colons instruction.
• Chain-of-thought prompting (P5) did not improve originality across models.

Limitations

• Single creative task (plastic bottle AUT item), limiting generalization.
• Model factor confounds architecture, alignment, default parameters and serving infrastructure.
• Linear mixed model used on count data for fluency rather than a GLMM.

Experimental setup

Datasets: Alternative Uses Task (plastic bottle item)

Benchmarks: Alternative Uses Task (plastic bottle)

Metrics: originality (1-5 automated AUT score) fluency (count of ideas) ICC variance share uniqueness intra-model repetition rate thematic entropy

Builds on: Alternative Uses Task DIPPER INSTINCT GEPA

Closest papers by meaning

• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting (ICLR 2024 · similarity 0.89)
• IDEAFix: Evaluation Framework for Creative Defixation Prompting in LLMs (NeurIPS 2026 · similarity 0.89)
• Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.88)
• Emergent Response Planning in LLMs (ICML 2025 · similarity 0.87)
• Evaluating Large Language Models at Evaluating Instruction Following (ICLR 2024 · similarity 0.87)
• Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.86)