Research map / Large language models / Agents, code and math
RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficulty
Xingjian Hu, Ziqian Zhang, Yue Huang, Kai Zhang, Ruoxi Chen, Yixin Liu, Qingsong Wen, Kaidi Xu, Xiangliang Zhang, Neil Zhenqiang Gong, Lichao Sun
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use EIP difficulty instead of simple error-count rank for question difficulty | question difficulty estimation · Human difficulty alignment | consensus agreement (%): 62.9 -> 90.0 | Improved |
| use EIP instead of 1PL/2PL/Multi-IRT | question difficulty estimation · Human difficulty alignment | consensus agreement (%): 52.9 -> 90.0 (vs Multi-IRT) | Improved |
| use a mixed-size model pool instead of a single-size pool (ablation) | EIP · Human difficulty alignment | consensus agreement (%): 38.6%-64.3% -> 90.0% | Improved |
| use a mixed-size model pool instead of homogeneous pools (ablation) | EIP · difficulty distribution | dead nodes: reduced by 83%; extreme cases below 3% | Improved |
| replace IRT with EIP | difficulty/competency estimation · Convergence time | time (s): 1782.75 -> 0.00597 (vs 1PL IRT) | Improved |
| remove half of the model pool (15 of 30) (ablation) | EIP · Question difficulty rank stability | Spearman rho: 0.9978 (k=1) -> 0.9382 (k=15) | Mixed |
| drop one dataset (HellaSwag or MMLU-Pro) from the pool (ablation) | EIP model ranking · Leave-one-dataset-out | Spearman rho: 0.9604 / 0.9626 (vs 0.9973 for small datasets) | No clear effect |
| weight correct answers by question difficulty (EIP) instead of plain accuracy/IRT in simulation | model ranking · Simulated 5 models, 100 questions | agreement with ground-truth ranking: only EIP matched both orderings | Improved |
| use open-weight model pool only for difficulty estimation (ablation) | EIP · Difficulty correlation with full pool | Spearman: 0.96 (open-weight) vs 0.78 (proprietary) | No clear effect |
Closest papers by meaning
- Reliable and Efficient Amortized Model-based Evaluation (ICML 2025 · similarity 0.92)
- RepLiQA: A Question-Answering Dataset for Benchmarking LLMs on Unseen Reference Content (NeurIPS 2024 · similarity 0.91)
- Rethinking Generative Large Language Model Evaluation for Semantic Comprehension (ICML 2024 · similarity 0.91)
- PiCO: Peer Review in LLMs based on Consistency Optimization (ICLR 2025 · similarity 0.90)
- Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages (AAAI 2025 · similarity 0.90)
- A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answering (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
