Research map / Label-efficient and robust learning / Knowledge distillation
Computerized Adaptive Testing via Collaborative Ranking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Collaborative Computerized Adaptive Testing (CCAT), which uses stored collaborative students as ranking anchors for question selection and ability estimation to improve ranking consistency in CAT.
Problem
Standard CAT optimizes ability estimation accuracy, but high-stakes exams need correct student ranking; independent per-student testing makes ranking consistency hard to optimize.
Approach
Treat fully-answered collaborative students as anchors: select questions that separate the tested student from collaborative students of similar ability, and define the final score as the fraction of collaborative students the tested student outranks, with a Hoeffding-based guarantee on ranking consistency.
Key results
- On NIPS-EDU with MCMC-IRT at step 5, CCAT reaches 0.8149 intra-class ranking consistency versus 0.7982 for CCAT without collaborative estimation and 0.7912 for the best baseline FSI (0.7411 for Random).
- On JUNYI with MCMC-IRT at step 5, CCAT reaches 0.8448 versus 0.8212 for FSI and 0.6527 for Random.
- The authors report an average 5% rise in ranking consistency over other methods, with larger gains for short tests; ACC and AUC stay comparable to other selectors.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add collaborative ability estimation (ablation) | CCAT with IRT-GD · NIPS-EDU, 5 steps | intra-class ranking consistency: 0.7320 -> 0.7533 | Helped |
| add collaborative ability estimation (ablation) | CCAT with IRT-GD · JUNYI, 5 steps | intra-class ranking consistency: 0.8026 -> 0.8092 | Helped |
| add collaborative ability estimation (ablation) | CCAT with IRT-MCMC · NIPS-EDU, 5 steps | intra-class ranking consistency: 0.7982 -> 0.8149 | Helped |
| add collaborative ability estimation (ablation) | CCAT with IRT-MCMC · JUNYI, 5 steps | intra-class ranking consistency: 0.8190 -> 0.8448 | Helped |
| use CCAT instead of FSI | IRT-MCMC question selection · NIPS-EDU, 20 steps | intra-class ranking consistency: 0.8975 -> 0.8969 | No clear effect |
| use CCAT instead of FSI | IRT-MCMC question selection · JUNYI, 20 steps | intra-class ranking consistency: 0.9257 -> 0.9273 | Helped |
| use CCAT instead of NCAT | IRT-GD question selection · NIPS-EDU, 5 steps | intra-class ranking consistency: 0.7611 -> 0.7533 | Hurt |
| use CCAT instead of NCAT | IRT-GD question selection · JUNYI, 5 steps | intra-class ranking consistency: 0.5198 -> 0.8092 | Helped |
Surprising findings
- BECAT, the state-of-the-art accuracy method, performs near random selection on ranking consistency (Figure 1b).
- Accurate ability estimation does not guarantee correct ranking; minimizing MSE can still yield inconsistent rankings.
Limitations
- Performance degrades with longer test sequences, according to the authors.
- Ranking-consistency estimates rely on approximations when collaborative students have not answered all questions, and on sigmoid surrogates of outperform indicators.
- Gradient-descent IRT estimation underperforms MCMC for ranking consistency.
Experimental setup
Datasets: NIPS-EDU JUNYI
Benchmarks: NIPS-EDU JUNYI
Metrics: Kendall-based intra-class ranking consistency inter-class ranking consistency AUC ACC NDCG
Baselines: Random FSI KLI MAAT BOBCAT NCAT BECAT
Builds on: Item Response Theory BOBCAT NCAT BECAT
Closest papers by meaning
- A Unified Adaptive Testing System Enabled by Hierarchical Structure Search (ICML 2024 · similarity 0.86)
- Explicit and Implicit Examinee-Question Relation Exploiting for Efficient Computerized Adaptive Testing (AAAI 2025 · similarity 0.86)
- An Autoencoder-Like Nonnegative Matrix Co-Factorization for Improved Student Cognitive Modeling (NeurIPS 2024 · similarity 0.79)
- Enhancing Cognitive Diagnosis Using Un-interacted Exercises: A Collaboration-Aware Mixed Sampling Approach (AAAI 2024 · similarity 0.79)
- EduDiag: A Benchmark for Educational Diagnostic Reasoning with Error Tracing and Correction on Large Multimodal Models (CVPR 2026 · similarity 0.78)
- PT-MoE: An Efficient Finetuning Framework for Integrating Mixture-of-Experts into Prompt Tuning (NeurIPS 2025 · similarity 0.78)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.