Research map / Large language models / Language, safety and interpretability
Is Automated Topic Model Evaluation Broken? The Incoherence of Coherence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Wikipedia reference corpus for NPMI instead of in-domain NYT reference corpus | NPMI · NYT topics, ratings | Spearman correlation with human ratings: 0.37 -> 0.48 | Helped |
| use NPMI model ranking instead of human judgment to pick a winning topic model | Dirichlet-VAE vs Gibbs-LDA · Wiki and NYT | model ranking | Mixed |
| use a small held-out validation set as NPMI reference corpus instead of training corpus | NPMI · Wiki topics, intrusion | Spearman correlation with human intrusion: 0.39 -> 0.17 | Hurt |
| filter out respondents unfamiliar with topic words (ablation) | D-VAE human evaluation · intrusion and ratings | mean human scores | Helped |
| use best automated metric (Cv, train reference) to claim model differences | automated coherence · Concatenated ratings | false discovery rate (%): 19 | Hurt |
| use Cv with 110-token window instead of NPMI with 10-token window | automated coherence · Concatenated ratings, train reference | false discovery rate (%): 37 -> 19 | Helped |
| higher automated coherence of a topic | NPMI/Cv · topic familiarity | familiar-word share: Pearson rho = -0.29 | Hurt |
Closest papers by meaning
- BBScore: A Brownian Bridge Based Metric for Assessing Text Coherence (AAAI 2024 · similarity 0.87)
- SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks (NeurIPS 2025 · similarity 0.83)
- Maximizing Signal in Human-Model Preference Alignment (AAAI 2025 · similarity 0.83)
- Handling Missing Responses under Cluster Dependence with Applications to Language Model Evaluation (NeurIPS 2025 · similarity 0.83)
- Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge (ICLR 2025 · similarity 0.82)
- Fine-Tuning Language Models for Factuality (ICLR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.