Research map / Learning theory and algorithms / Prediction and decision losses
A Unified Adaptive Testing System Enabled by Hierarchical Structure Search
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UATS, a data-driven framework that unifies computerized adaptive testing and multistage testing as a differentiable hierarchical test structure search, with convergence and gradient-error guarantees.
Problem
Adaptive testing systems are split into CAT (one question per step) and MST (testlets per step, expert-designed), with no unified framework that automatically chooses both question count and composition per step.
Approach
Relaxes the discrete choice of questions at each test step into softmax weights over the question bank and optimizes them by bilevel gradient descent (inner IRT ability estimate, outer query-set loss), with a greedy layer-wise search that unifies CAT and MST.
Key results
- On ASSIST with IRT, UATS improves AUC@20 by about 1.31% relative to the best prior SOTA on average, and ACC@20 by 0.71%.
- In the ability-estimation simulation on NIPS-EDU, UATS reaches the same estimation error as the SOTA with about 20% fewer questions on average.
- Against MST baselines built with ATA-generated testlets on NIPS-EDU, UATS improves ACC by 2.7% and AUC by 3.1% on average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UATS instead of KLI | IRT-based adaptive testing · ASSISTments | accuracy@20: 66.75 -> 67.07 | Helped |
| Use UATS instead of NCAT | IRT-based adaptive testing · ASSISTments | AUC@20: 67.88 -> 68.00 | Helped |
| Use UATS instead of BOBCAT | NeuralCDM adaptive testing · ASSISTments | accuracy@20: 68.20 -> 68.32 | Helped |
| Use UATS instead of BOBCAT | NeuralCDM adaptive testing · ASSISTments | AUC@20: 68.05 -> 68.92 | Helped |
| Use UATS instead of BOBCAT | IRT-based adaptive testing · NIPS-EDU | accuracy@20: 67.51 -> 68.17 | Helped |
| Use UATS instead of KLI | IRT-based adaptive testing · NIPS-EDU | AUC@20: 73.57 -> 74.85 | Helped |
| Use UATS instead of BOBCAT | NeuralCDM adaptive testing · NIPS-EDU | accuracy@20: 69.46 -> 70.15 | Helped |
| Use UATS instead of NCAT | NeuralCDM adaptive testing · NIPS-EDU | AUC@20: 76.34 -> 76.86 | Helped |
| Use UATS instead of BOBCAT | IRT-based adaptive testing · EXAM | accuracy@20: 73.80 -> 74.51 | Helped |
| Use UATS instead of BOBCAT | IRT-based adaptive testing · EXAM | AUC@20: 77.32 -> 78.63 | Helped |
| Use UATS instead of NCAT | NeuralCDM adaptive testing · EXAM | accuracy@20: 74.21 -> 74.53 | Helped |
| Use UATS instead of BOBCAT | NeuralCDM adaptive testing · EXAM | AUC@20: 77.58 -> 77.98 | Helped |
Surprising findings
- Implicit selection methods such as NCAT perform better than UATS in the earliest test steps before UATS overtakes them after about step 4.
Limitations
- MST experiments rely on testlets generated by ATA because the real MST dataset is confidential, so the MST comparison is not on expert-designed testlets.
- The convergence and gradient-error guarantees depend on IRT's strong convexity assumption and the paper states the inner-loop approximation introduces error.
- Fairness across students receiving different questions is explicitly left out of scope.
Experimental setup
Datasets: ASSISTments NIPS-EDU EXAM
Benchmarks: ASSISTments NIPS-EDU EXAM
Metrics: ACC AUC mean squared error of ability estimate
Baselines: Random FSI KLI MAAT BECAT BOBCAT NCAT MST (ATA-generated testlets)
Builds on: Item Response Theory NeuralCDM NCAT BOBCAT BECAT
Closest papers by meaning
- Computerized Adaptive Testing via Collaborative Ranking (NeurIPS 2024 · similarity 0.86)
- Explicit and Implicit Examinee-Question Relation Exploiting for Efficient Computerized Adaptive Testing (AAAI 2025 · similarity 0.85)
- Active Test-Time Adaptation: Theoretical Analyses and An Algorithm (ICLR 2024 · similarity 0.81)
- Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks (ICML 2026 · similarity 0.81)
- Automated Evaluation of Retrieval-Augmented Language Models with Task-Specific Exam Generation (ICML 2024 · similarity 0.80)
- Context-Aware Testing: A New Paradigm for Model Testing with Large Language Models (NeurIPS 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.