Research map / Learning theory and algorithms / Prediction and decision losses
Comprehensive Algorithm Portfolio Evaluation using Item Response Theory
Sevvandi Kandanaarachchi, Kate Smith-Miles
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| relax strength tolerance epsilon from 0 to 0.01 (ablation) | AIRT strengths · CSP-Minizinc-2016 | latent trait occupancy of LCG-Glucose-UC-free: 0.717 -> 0.828 | Improved |
| relax strength tolerance epsilon from 0 to 0.01 (ablation) | AIRT strengths · CSP-Minizinc-2016 | number of algorithms with strengths: 5 -> 7 | Improved |
| relax strength tolerance epsilon from 0 to 0.01 (ablation) | AIRT strengths · CSP-Minizinc-2016 | latent trait occupancy of LCG-Glucose-free: 0.071 -> 0.111 | Improved |
| relax strength tolerance epsilon from 0 to 0.01 (ablation) | AIRT strengths · CSP-Minizinc-2016 | latent trait occupancy of OR-Tools-free: 0.071 -> 0.071 | No clear effect |
| use the AIRT model to fit HaifaCSP-free instead of other algorithms | IRT meta-model · CSP-Minizinc-2016 | MSE / |AUAEC - AUPEC|: 0.100 and 0.224 | Hurt |
| use airt portfolio selection instead of topset and Shapley | algorithm portfolio selection · MAXSAT-PMS-2016 (10-fold CV) | mean performance gap | Improved |
| use airt portfolio selection instead of topset and Shapley | algorithm portfolio selection · ASP-POTASSCO (10-fold CV) | mean performance gap | No clear effect |
| use airt portfolio selection instead of topset and Shapley | algorithm portfolio selection · SAT11-INDU (10-fold CV) | mean performance gap | Hurt |
Closest papers by meaning
- Can We Trust Item Response Theory for AI Evaluation? (NeurIPS 2026 · similarity 0.88)
- IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation (NeurIPS 2024 · similarity 0.83)
- Computerized Adaptive Testing via Collaborative Ranking (NeurIPS 2024 · similarity 0.82)
- Foundations of the Theory of Performance-Based Ranking (CVPR 2025 · similarity 0.82)
- AI Evaluation Should Require Standardized Item-Level Data Releases (NeurIPS 2026 · similarity 0.82)
- Unveiling the Tapestry of Automated Essay Scoring: A Comprehensive Investigation of Accuracy, Fairness, and Generalizability (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
