Research map / Large language models / Language, safety and interpretability
Are Key-Phrases All That Reviewers Care About? A Comprehensive Benchmarking of Reviewer Matchmaking Systems
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Benchmarks six document-representation reviewer matchmaking models against fourteen key-phrase-extraction-based models on the CMU-RM23 dataset, arguing standard correlation measures should replace Precision@K and Kendall Loss.
Problem
Reviewer matchmaking systems are far from accurate, and existing evaluation measures (Precision@K, Kendall Loss) are unreliable for judging how well relevance scores align with reviewer confidence.
Approach
Applies key-phrase extraction models to build reviewer matchmaking pipelines (KPE-RM), compares them with SOTA document-representation RM models, and replaces Kendall Loss with Pearson, Spearman and Kendall correlations between mean predicted and mean reviewer confidence scores.
Key results
- PatternRank-RM (Jaccard, mean, k=15) reaches Pearson 0.433, Spearman 0.423, Kendall 0.339 on title+abstract, beating the best DR-RM SPECTER2-RM (0.425, 0.417, 0.296).
- MultipartiteRank-RM is reported to top all correlations on full-text input.
- Kendall Loss shows strong negative inter-correlation with standard correlation measures (e.g., Spearman -0.927 vs Pearson on title+abstract), supporting its replacement.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PatternRank-RM instead of SPECTER2-RM | CMU-RM23 reviewer matchmaking · CMU-RM23 title + abstract | Pearson r: 0.425 -> 0.433 | Helped |
| Use PatternRank-RM instead of SPECTER2-RM | CMU-RM23 reviewer matchmaking · CMU-RM23 title + abstract | Spearman ρ: 0.417 -> 0.423 | Helped |
| Use PatternRank-RM instead of SPECTER2-RM | CMU-RM23 reviewer matchmaking · CMU-RM23 title + abstract | Kendall τ: 0.296 -> 0.339 | Helped |
| Use embedding-based similarity rather than GloVe cosine similarity (ablation) | KPE-RM models · CMU-RM23 | RM performance | Helped |
| Use cosine-SBERT similarity rather than Jaccard similarity (ablation) | KPE-RM models · Full-text CMU-RM23 | RM performance | Helped |
| Switch aggregation mode between max and mean (ablation) | KPE-RM models · CMU-RM23 | RM performance | No clear effect |
| Use top-5 key phrases rather than top-15 or top-30 (ablation) | KPE-RM models · CMU-RM23 | RM performance | Hurt |
| Vary the number of extracted key phrases between 15 and 30 (ablation) | KPE-RM models · CMU-RM23 | RM performance | No clear effect |
| Apply sub-word elimination (ablation) | KPE-RM models · CMU-RM23 | RM performance | Hurt |
| Change the order of sub-word elimination and top-k selection (ablation) | KPE-RM models · CMU-RM23 | RM performance | No clear effect |
| Use full text rather than title + abstract (ablation) | KPE-RM models · CMU-RM23 | RM performance | No clear effect |
| Use KPE models without contextualizing them to the reviewer matchmaking task | KPE-RM models · CMU-RM23 | RM accuracy | Hurt |
Surprising findings
- Kendall Loss-based leaderboards are strongly negatively correlated with standard correlation leaderboards, so the prior benchmark measure ranks models inversely.
- Top KPE models on classical keyphrase extraction benchmarks do not rank similarly on the reviewer matchmaking task, so KPE models cannot be used plug-and-play.
Limitations
- Best KPE-RM models reach only moderate rank correlation, since keyphrase extraction is not optimized for the RM task.
- Evaluation uses a single gold-standard dataset (CMU-RM23) with 463 manuscripts and 58 reviewers.
- Sub-word elimination and full-text inputs showed no consistent benefit, leaving the utilization of RM-relevant content open.
Experimental setup
Datasets: CMU-RM23 gold standard dataset Inspec Krapivin NUS SemEval 2017
Benchmarks: CMU-RM23
Metrics: Pearson r Spearman rho Kendall tau Kendall Loss Precision@K
Baselines: SPECTER2-RM SPECTER2 + SciNCL-RM SciNCL-RM SPECTER-RM SPECTER+MFR-RM BM25-RM
Builds on: SPECTER2 SciNCL KeyBERT PatternRank PromptRank CMU-RM23 gold standard dataset
Closest papers by meaning
- From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review (NeurIPS 2025 · similarity 0.86)
- Allocate Marginal Reviews to Borderline Papers Using LLM Comparative Ranking (NeurIPS 2026 · similarity 0.82)
- SciArena: An Open Evaluation Platform for Non-Verifiable Scientific Literature-Grounded Tasks (NeurIPS 2025 · similarity 0.82)
- RevisEval: Improving LLM-as-a-Judge via Response-Adapted References (ICLR 2025 · similarity 0.82)
- PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistencies (ICLR 2026 · similarity 0.81)
- Mixture of Knowledge Minigraph Agents for Literature Review Generation (AAAI 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.