Research map / Large language models / Reasoning and chain of thought

TokenPowerBench: Benchmarking the Power Consumption of LLM Inference

AAAI 2026 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces TokenPowerBench, a lightweight open-source benchmark that measures GPU/CPU/DRAM/node power and attributes LLM inference energy to prefill and decode phases across models, engines, batch sizes, context lengths, parallelism and quantization.

Problem

Existing benchmarks (MLPerf Power, etc.) do not systematically measure end-to-end power of LLM inference, normalize energy per token, cover frontier-scale models, or explore inference configuration space without expensive external meters.

Approach

A declarative configuration layer plus a measurement layer that combines vendor telemetry (NVML/DCGM, RAPL, IPMI) with phase-tagged power samples, so energy is attributed to prefill and decode per request and reported as joules per token, without specialized power meters.

Key results

• Moving LLaMA-3 from 1B to 70B parameters increases energy per token by 7.3x despite a 70x parameter increase.
• Pure tensor parallelism (TP 16, PP 1) gives the best energy efficiency for Llama 3 405B, Falcon 180B and Qwen 480B on 16 H100 GPUs across all workloads.
• FP8 weight quantization for Llama 3 405B cuts energy per token by roughly 30% relative to FP16 across three workload profiles.

Surprising findings

• Pure tensor parallelism is consistently most energy-efficient, because long pipelines leave some GPUs idle.
• TensorRT-LLM and vLLM use roughly 3x more prefill energy than DeepSpeed and Transformers, yet reduce energy per token by 25-40% versus Transformers.

Limitations

• Evaluation limited to a single H100 cluster; other GPU architectures planned as future work.
• Accuracy-energy trade-off not yet quantified; quantization accuracy only noted informally on a prompt set.

Experimental setup

Datasets: Alpaca LongBench

Benchmarks: Alpaca LongBench

Metrics: joules per token prefill energy decode energy per token energy per response throughput tokens/s peak power

Baselines: MLPerf Power LLM-Inference-Bench Samsi et al.

Builds on: MLPerf Power NVML RAPL vLLM

Closest papers by meaning

• The Silent Hyperparameter: Quantifying the Impact of Inference Backends on LLM Reproducibility (NeurIPS 2026 · similarity 0.91)
• Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.90)
• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.90)
• Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.90)
• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.89)
• D-LLM: A Token Adaptive Computing Resource Allocation Strategy for Large Language Models (NeurIPS 2024 · similarity 0.89)