Research map / Large language models / Reasoning and chain of thought

Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Item Response Scaling Laws (IRSL), integrating IRT (incl. a Beta-likelihood variant, Beta-IRT) into pre-training downstream and test-time scaling law estimation to cut evaluation cost from O(M×N) to O(M+N).

Problem

Deriving neural scaling laws for downstream performance and test-time repeated sampling requires prohibitively many model-question evaluations; traditional per-LM-per-benchmark fits do not generalize and binary responses waste information.

Approach

Factorize LM ability from question parameters using IRT (1PL/2PL), model scaling via ability as a function of log-compute or sample count, and use a Beta likelihood on empirical probability responses (token probabilities or pass rates) so that few test takers suffice for calibration; ability transfers across benchmarks with the same measurement objective.

Key results

• Beta-IRT achieves near-perfect 1PL parameter recovery (RMSE < 0.05, rho > 0.999) with as few as 2 test takers, versus Binary-IRT needing about 64 in simulation.
• Pre-training IRSL uses only 50 questions per benchmark (a 99.9% reduction) for comparable or superior decision accuracy; Beta-IRT matches traditional Pcc on ARC Challenge, ARC Easy, and MMLU and wins on several lower-quality benchmarks, but fails on BoolQ, HellaSwag, WinoGrande.
• Test-time IRSL yields lower MAE for -log pass@k than traditional scaling across nearly all LM-benchmark pairs on 100 random splits; cross-benchmark ability transfer from AIME 2024 to AIME 2025 works with theta correlation 0.80.

Surprising findings

• Beta-IRT fails to capture a predictive trend on extremely homogeneous benchmarks such as BoolQ, HellaSwag, and WinoGrande.
• Test-time evaluation with only 12 LMs and 120 questions gives correlations for cross-benchmark ability transfer that are weaker than pre-training ones.

Limitations

• Requires pre-calibrated item parameters from prior model responses, limiting use to established benchmarks and specific evaluation setups.
• Test-time experiments are small in scale (12 LMs, 120 questions, 4 benchmarks).
• Benchmark homogeneity leads to failure on noisy benchmarks where neither approach yields reliable trends.

Experimental setup

Datasets: DataDecide ARC Easy ARC Challenge MMLU BoolQ CommonsenseQA HellaSwag OpenBookQA PIQA SocialIQA WinoGrande AIME 2024

Benchmarks: ARC Easy ARC Challenge MMLU BoolQ CommonsenseQA HellaSwag OpenBookQA PIQA SocialIQA WinoGrande AIME 2024 AIME 2025

Metrics: Decision Accuracy RMSE Pearson correlation MAE of scaling curve

Baselines: Traditional scaling law (Acc) Traditional scaling law (Pcc) Binary-IRT 1PL Binary-IRT 2PL Traditional test-time power-law scaling

Builds on: Item Response Theory (1PL/2PL) DataDecide beta^3-IRT pass@k unbiased estimator Binary-IRT

Closest papers by meaning

• A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.91)
• Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.89)
• On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.89)
• A Simple Model of Inference Scaling Laws (ICML 2025 · similarity 0.89)
• Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.89)
• Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.89)