Research map / Large language models / Language, safety and interpretability

The Subjectivity of Monoculture

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: Bias measurement and mitigation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Argues that monoculture claims are relative to a chosen null model and population; uses multidimensional and 1D IRT nulls on HELM, Open LLM Leaderboard and ACSIncome to show excess residual correlation shrinks with item difficulty and depends on model population.

Problem

Measuring whether models agree 'too much' (monoculture) requires a baseline null of independence, but the choice of null model and the population of models and items strongly affect conclusions about excess agreement.

Approach

Frames monoculture as a comparative inference relative to a null ladder of latent-variable models (IRT); shows that adding item difficulty (and more dimensions) absorbs residual correlation, and that inferences are unstable when the model population is homogeneous, formalized with a heterogeneity measure on latent factors.

Key results

• On HELM and HF, mean absolute pairwise residual correlation shrinks toward zero as IRT dimension K grows to 64.
• Under 1D IRT with item difficulty (IRT-1), residual correlations are substantially attenuated relative to Kim et al. and Goel et al. baselines, with some pairs flipping from strongly positive to slightly negative.
• On ACSIncome and HELM, restricting to homogeneous model populations (e.g., only RF or only OpenAI models) makes excess correlation inferences unstable; adding diverse models shifts inferred difficulty distributions.

Surprising findings

• Monoculture virtually disappears on one dataset once item difficulty is included in the null.
• Some model pairs flip from strongly positive correlation to slightly negative under the item-difficulty null.

Limitations

• Null model choice (IRT) is itself subjective and dataset-agnostic; alternatives such as topic specialization are not explored.
• Experiments use binary correctness, discarding answer-choice information and logits.
• Theory relies on regularity conditions and the heterogeneity result is local and asymptotic.

Experimental setup

Datasets: HELM (MMLU) Open LLM Leaderboard (HF) ACSIncome

Benchmarks: HELM Open LLM Leaderboard ACSIncome

Metrics: mean squared error absolute mean pairwise residual correlation AbsMax median absolute residual correlation

Baselines: Kim et al. monoculture measure Goel et al. CAPA IRT-0.5 (1D IRT without item difficulty)

Builds on: item response theory multidimensional IRT CAPA Kim et al. algorithmic monoculture measure

Closest papers by meaning

• Generative Monoculture in Large Language Models (ICLR 2025 · similarity 0.90)
• CultureLLM: Incorporating Cultural Differences into Large Language Models (NeurIPS 2024 · similarity 0.86)
• CulturePark: Boosting Cross-cultural Understanding in Large Language Models (NeurIPS 2024 · similarity 0.85)
• Attributing Culture-Conditioned Generations to Pretraining Corpora (ICLR 2025 · similarity 0.85)
• Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Function (ICML 2024 · similarity 0.85)
• Monoculture in Matching Markets (NeurIPS 2024 · similarity 0.85)