Research map / Large language models / Reasoning and chain of thought

Cost-of-Pass: An Economic Framework for Evaluating Language Models

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes cost-of-pass, an economics-grounded metric (expected dollar cost per correct solution) and a frontier version including a human-expert baseline, and uses it to track LM cost-efficiency progress and assess inference-time techniques on math, knowledge, and bias benchmarks.

Problem

Existing LM evaluations report accuracy or cost separately, or use fixed budgets or heuristic proxies, so they do not capture the economic value of model performance relative to inference cost and human labor.

Approach

Adapts Farrell's production-efficiency and stochastic frontier theory to LMs: cost-of-pass = expected cost per attempt divided by success probability, and the frontier cost-of-pass takes the minimum over available LMs and a estimated human-expert cost, enabling tracking of progress, counterfactual family essentialness, and evaluation of inference-time techniques.

Key results

• Frontier cost-of-pass for MATH-500 halved approximately every 2.6 months and for AIME-2024 every 7.1 months (May 2024 to Feb 2025).
• Lightweight models are essential for basic quantitative tasks, large models for knowledge-based (GPQA), and reasoning models for complex quantitative tasks.
• Majority voting yields 0% relative gain on all datasets; self-refinement gives 24.9% gain on GPQA-Diamond; TALE-EP gives gains on several tasks, e.g. 66.6% on GSM8K.

Surprising findings

• Majority voting gave zero economic gain across all tested models and datasets despite potentially improving accuracy.
• Human experts remain fully essential for GSM8K, GPQA-Diamond, MATH-500, and AIME-2024 under the essentialness metric.

Limitations

• Human-expert cost estimates rely on assumed hourly rates and time per question derived from secondary sources, and assume near-perfect expert accuracy.
• Evaluation uses up to 128 instances per dataset with 8 runs per model, so estimates are sample-limited.
• Results depend on per-token API prices, which change over time and differ by provider.

Experimental setup

Datasets: Two-Digit Addition GSM8K BBQ GPQA-Diamond MATH-500 AIME 2024 Tau-bench

Benchmarks: Two-Digit Addition GSM8K BBQ GPQA-Diamond MATH-500 AIME 2024 Tau-bench

Metrics: cost-of-pass frontier cost-of-pass accuracy dollar cost per attempt relative gain essentialness

Baselines: GPT-4o GPT-4o mini Llama-3.1-8B Llama-3.3-70B Llama-3.1-405B Claude Sonnet-3.5 OpenAI o1-mini OpenAI o1 DeepSeek-R1 OpenAI o3-mini

Builds on: Farrell's productive efficiency theory stochastic frontier production functions self-consistency TALE

Closest papers by meaning

• A Theory of Training Profit-Optimal LLMs (NeurIPS 2026 · similarity 0.90)
• Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.87)
• The Shadow Price of Reasoning: Economic Perspective on Optimal Budget Allocation for LLMs (ICML 2026 · similarity 0.87)
• STEER: Assessing the Economic Rationality of Large Language Models (ICML 2024 · similarity 0.87)
• Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Models (NeurIPS 2025 · similarity 0.87)
• Tina: Tiny Reasoning Models via LoRA (ICLR 2026 · similarity 0.87)