Research map / Large language models / AI and society
Impacts of Aggregation on Model Diversity and Consumer Utility
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical study of how choice-based aggregation (Bradley-Terry-Luce routing/consumer choice) affects consumer welfare and producer incentives, showing non-monotonicity and that a "weighted winrate" objective reduces homogenization relative to winrate, with experiments on LLMRouterBench, MT-Bench-101, and WildBench.
Problem
Under noisy choice-based aggregation of multiple AI models (consumers or routers picking models per task), adding or improving models may not raise consumer welfare, and producers optimizing winrate tend to homogenize their models rather than specialize.
Approach
Model aggregation under Bradley-Terry-Luce choice is analyzed for consumer welfare; producers optimizing winrate homogenize, while a proposed weighted winrate objective (winrate times the value delivered) incentivizes specialization that aligns with consumer welfare.
Key results
- Winrate-maximizing producers equalize winrate across tasks (about 0.97 winrate on LLMRouterBench allocation), whereas weighted winrate and consumer welfare both specialize on 8 of 14 tasks.
- BTL aggregation is non-monotone: in the example, raising a model's value from 1 to 4 lowers the set value from about 4.92 to 4.73 (beta=1).
- Weighted winrate gives better producer incentives than winrate for almost all model pairs in LLMRouterBench replacement experiments, though not always.
Surprising findings
- Adding a model that is weaker than every existing model on average can increase consumer welfare, and adding a stronger model can decrease it under BTL aggregation.
- Increasing the value of a single model can reduce consumer welfare under BTL aggregation, a non-monotonicity shown to be unavoidable for any separable choice function.
Limitations
- Producer abstention cannot be incentivized by weighted winrate; producers are indifferent between returning value 0 and abstaining.
- Assumes deterministic per-task values and BTL choice model; experiments use benchmark scores as value proxies.
- Weighted winrate requires eliciting real-valued user value, which users may report inaccurately.
Experimental setup
Datasets: LLMRouterBench MT-Bench-101 WildBench
Benchmarks: LLMRouterBench MT-Bench-101 WildBench
Metrics: consumer welfare winrate weighted winrate
Baselines: winrate objective consumer welfare optimum
Builds on: Bradley-Terry model Colonel Blotto game Tullock contest LLMRouterBench
Closest papers by meaning
- Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation (NeurIPS 2026 · similarity 0.85)
- Pricing and Competition for Generative AI (NeurIPS 2024 · similarity 0.84)
- Mechanism Design for LLM Fine-tuning with Multiple Reward Models (NeurIPS 2025 · similarity 0.84)
- EcoGym: Evaluating LLMs for Long-Horizon Plan-and-Execute in Interactive Economies (NeurIPS 2026 · similarity 0.84)
- Routing, Cascades, and User Choice for LLMs (ICLR 2026 · similarity 0.84)
- Model Alignment as Prospect Theoretic Optimization (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.