Research map / Large language models / Language, safety and interpretability

A Simple Model of Inference Scaling Laws

ICML 2025 · Large language models › Language, safety and interpretability · approach: LLM pruning and scaling data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a toy memorization-based statistical model that explains power-law scaling of pass@k (coverage) with repeated inference attempts, defines an inference loss, links it to inference compute, and validates on a VAE reconstruction task.

Problem

Understanding and predicting how performance (coverage/pass@k) of generative models improves with the number of repeated inference attempts, and how this trades off against inference cost.

Approach

Model per-sample failure probabilities as Beta-distributed, so averaging p^k over samples yields a power-law pass@k in k with exponent set by Beta parameters; additionally model trial correlations via an effective number of independent trials k_eff = H_k(kappa) that decays power-law, and map k to a FLOPS-based inference cost.

Key results

• Beta-distributed failure probabilities give pass@k ~ A(1 - Gamma(beta) k^(-beta)/B(alpha,beta)) and an inference loss decaying as a power law in k with exponent beta.
• Using typical fitted values beta~0.35 for LLM pass@k curves, the paper argues most problems are hard for the model while easy samples are covered within about 100 trials.
• On a VAE reconstruction task (Fashion-MNIST, T=1.1), the effective-k model approximates empirical pass@k curves, and trial-error correlation eigenvalues decay as a power law.

Measured effects reported in the paper

Limitations

• Toy model; the authors state many other models could yield the same functional form, so it does not uniquely identify mechanisms.
• Effective-k approach is described as phenomenological and kappa is fit from data.
• Validation on LLMs relies on previously reported pass@k curves rather than new LLM experiments; the cost mapping uses illustrative numbers.

Experimental setup

Datasets: Fashion-MNIST

Benchmarks: Fashion-MNIST VAE reconstruction

Metrics: pass@k coverage normalized reconstruction error correlation matrix eigenvalues

Builds on: Hutter model of memorization and generalization neural scaling laws repeated sampling inference scaling

Closest papers by meaning

• Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.91)
• Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.91)
• On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.90)
• A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.90)
• Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.90)
• Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.90)