Research map / Large language models / Instruction and fine-tuning

Generative Monoculture in Large Language Models

ICLR 2025 · Large language models › Instruction and fine-tuning · approach: Efficient LLM fine-tuning systems · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces "generative monoculture" (LLM output diversity narrowing relative to training-data distributions) and measures it on book review and code generation tasks across Llama-2, Vicuna, GPT-3.5, GPT-4, and Claude-3, finding that common sampling and prompting mitigations are insufficient.

Problem

LLM outputs do not faithfully represent the diversity of responses present in their training data, narrowing attribute distributions for tasks such as book reviews and code generation.

Approach

Defines generative monoculture as a dispersion shift from source (human-written) to model-generated data for task-specific attributes, provides a measurement pipeline (data curation, attribute extraction, dispersion metrics), and tests mitigations (temperature, top-p, temperature decay, persona prompts), conjecturing alignment (RLHF) as a root cause.

Key results

• On book reviews, GPT-3.5 and GPT-4 produce 100% of samples with average positivity in (0.95,1.00] under prompt (1), versus 2.8% for the source reviews.
• Llama-2-13b (pretrained) has 5.9% of samples in (0.95,1.00], much closer to the source than 44.7% for Vicuna-13b and 82.1% for Llama-2-13b-chat.
• LLM-generated code solutions are over 2x more accurate than average human solutions, but show high pairwise plagiarism scores while source solutions score zero.

Measured effects reported in the paper

Surprising findings

• Proprietary GPT-3.5 and GPT-4 show more severe monoculture than open-source Llama-family models.
• Higher temperature (T=1.5) and temperature decay fail to close the gap, and high randomness degrades output validity.

Limitations

• Full training corpora of the evaluated LLMs were not analyzed; Goodreads and CodeContests are proxies for training data.
• Attribute selection and extraction are subjective and sensitive to extractor reliability; LLM-based extraction (GPT-3.5) may be biased.
• Mitigating monoculture may allow toxic or harmful content to re-enter outputs.

Experimental setup

Datasets: Goodreads reviews CodeContests

Benchmarks: Goodreads reviews CodeContests SST-2

Metrics: mean sentiment entropy standard deviation mean pairwise Jaccard index mean pairwise cosine similarity mean pairwise fingerprint similarity accuracy unique word count valid generation fraction

Baselines: Llama-2-13b Vicuna-13b-v1.5 GPT-3.5-turbo-instruct GPT-4-turbo Claude-3-Sonnet

Builds on: RLHF nucleus sampling Winnowing BERTopic

Closest papers by meaning

• The Subjectivity of Monoculture (NeurIPS 2026 · similarity 0.90)
• Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond) (NeurIPS 2025 · similarity 0.87)
• Benchmarking LLMs' Judgments with No Gold Standard (ICLR 2025 · similarity 0.87)
• Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data (NeurIPS 2025 · similarity 0.87)
• Attributing Culture-Conditioned Generations to Pretraining Corpora (ICLR 2025 · similarity 0.87)
• Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testing (ICML 2025 · similarity 0.87)