Research map / Large language models / Language, safety and interpretability
Cheaply Estimating Inference Efficiency Metrics for Autoregressive Transformer Models
Deepak Narayanan, Keshav Santhanam, Peter Henderson, Rishi Bommasani, Tony Lee, Percy Liang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Send 10x more queries in parallel to the API (heavy load) | Anthropic/v4-s3 raw runtime · prompt size 512 | end-to-end runtime: up to 2x increase | Hurt |
| Use linear regression on a few calibration queries rather than exhaustive profiling | idealized runtime estimation · 4 HELM tasks | cost: 57x savings | Improved |
| Fit a linear model of runtime in output tokens per prompt size | open-source models on A100 · prompt sizes 1, 512, 1024, 1536 | R2: > 0.999 | Improved |
| Move from A100 to V100 GPUs | idealized cost · HELM tasks | runtime and cost | Hurt |
| Use idealized metrics instead of raw runtimes | OpenAI/davinci · HELM tasks Pareto frontier | Pareto frontier membership | Mixed |
Closest papers by meaning
- The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.87)
- TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (AAAI 2026 · similarity 0.85)
- Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.85)
- Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs (ICML 2024 · similarity 0.85)
- TimeBill: Time-Budgeted Inference for Large Language Models (AAAI 2026 · similarity 0.84)
- TEST-TIME SCALING IN DIFFUSION LLMS VIA HIDDEN SEMI-AUTOREGRESSIVE EXPERTS (ICLR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
