Research map / Large language models / Reasoning and chain of thought

Intelligence per Watt: Measuring Intelligence Efficiency of Local AI

NeurIPS 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measures local LM inference efficiency via a new metric, intelligence per watt (accuracy per unit power), across 20+ local models, 8-9 accelerators, and 1M+ queries from 2023 to 2025, and evaluates hybrid local-cloud routing savings.

Problem

Whether local inference with small models (≤20B active parameters) on local accelerators can viably absorb demand from centralized cloud LLM inference, and how efficiently local systems convert power into correct answers.

Approach

Defines intelligence per watt (IPW) and related per-joule metrics (IPJ, PPW, PPJ) that jointly capture model accuracy and accelerator power/latency, and uses them in a longitudinal profiling study with a cross-platform telemetry harness and a simulated local-cloud query router.

Key results

• Best local model per query (oracle-style routing) covers 88.7% of single-turn chat and reasoning queries, versus 71.3% for the best individual local model (gpt-oss-120b).
• Accuracy per watt improves 5.3x from 2023 to 2025 (7.92e-4 to 4.18e-3), decomposed into 3.1x model and 1.7x hardware gains.
• An 80%-accurate local-cloud router achieves 64.3% energy, 61.8% compute, and 59.0% cost savings versus routing all queries to Qwen3-235B on H200.

Surprising findings

• Cloud accelerators (NVIDIA B200, SambaNova SN40L) retain 1.4x to 7.4x per-query efficiency advantage over local Apple M4 Max at batch size 1.
• Best-of-local routing on reasoning benchmarks can exceed best-of-cloud coverage, attributed to best-of-N selection over many local models.

Limitations

• Accuracy for chat is a binary win/tie judgment against a Qwen3-235B reference, not ground truth.
• Power measured at accelerator subsystem with software telemetry that may carry 10-15% inaccuracy.
• Routing savings are simulated with assumed router accuracies rather than a deployed router.

Experimental setup

Datasets: WildChat NaturalReasoning MMLU-Pro SuperGPQA

Benchmarks: WildChat NaturalReasoning MMLU-Pro SuperGPQA

Metrics: accuracy per watt accuracy per joule perplexity per watt perplexity per joule coverage latency energy per query cost

Baselines: Qwen3-235B Claude Sonnet 4.5 Gemini 2.5 Pro GPT-5 Mixtral-8x7B Llama-3.1-8B

Builds on: RouteLLM MMLU-Pro SuperGPQA WildChat

Closest papers by meaning

• TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (AAAI 2026 · similarity 0.89)
• Bridging On-Device and Cloud LLMs for Collaborative Reasoning: A Unified Methodology for Local Routing and Post-Training (ICML 2026 · similarity 0.88)
• Cost-efficient Collaboration between On-device and Cloud Language Models (ICML 2025 · similarity 0.88)
• Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.86)
• MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving (ICML 2026 · similarity 0.86)
• Fast Inference for Augmented Large Language Models (NeurIPS 2025 · similarity 0.86)