Research map / Probabilistic and scientific modeling / Molecules and proteins
The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the ML.ENERGY Benchmark and Leaderboard for measuring service-aware inference energy of 40 generative AI models across 6 tasks on A100/H100 GPUs, with automated Pareto-based energy optimization recommendations.
Problem
Energy consumption of generative AI inference is poorly measured and understood; existing benchmarks either lack realistic serving conditions, fix batch size to 1, rely on TDP estimates, or are closed.
Approach
A benchmark that measures per-request inference energy under realistic serving (steady-state iteration-level batching, software GPU energy via Zeus) and builds time-energy Pareto frontiers to recommend the energy-optimal configuration satisfying a user latency constraint.
Key results
- Benchmarks 40 model architectures across 6 tasks; TDP-based energy estimates can overestimate energy by up to 4.1x (CodeGemma 2B on H100).
- Latency-aware Pareto optimization yields 44% energy reduction for Llama 3.1 8B at 100 ms TPOT target (77 ms configuration).
- For Stable Diffusion 2.1 with 5 s latency target, energy-optimal configuration at 3.63 s saves 21% energy versus latency-minimizing configuration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase max batch size from 4 to 64 (ablation) | Reasoning models · GPQA on NVIDIA H100 | energy per generation: 9713.7 -> 2770.8 | Helped |
| Increase max batch size from 4 to 64 (ablation) | Reasoning models · GPQA on NVIDIA H100 | energy per generation: 19974.4 -> 7595.4 | Helped |
| Increase max batch size from 4 to 64 (ablation) | Reasoning models · GPQA on NVIDIA H100 | energy per generation: 26419.7 -> 4520.6 | Helped |
| Increase max batch size from 4 to 64 (ablation) | Reasoning models · GPQA on NVIDIA H100 | energy per generation: 122523.1 -> 33096.4 | Helped |
| Increase maximum batch size and max batched tokens (ablation) | Llama 3.1 8B with chunked prefill · Synthetic long-context requests on H100 | energy per generation: 559.66 -> 151.50 | Helped |
| Increase max batched tokens beyond the energy-efficient range (ablation) | Llama 3.1 8B with chunked prefill · Synthetic long-context requests on H100 | energy per generation: 188.80; 191.61; 195.59; 191.88; 194.52 | Mixed |
| Use Swapping rather than Recomputation for preemption (ablation) | Mistral Nemo 12B · Overloaded inference server | energy per generation | Helped |
| Scale tensor parallelism from one GPU to two (ablation) | Llama 3.1 8B · Inference | energy consumption | No clear effect |
| Scale tensor parallelism beyond two GPUs (ablation) | Llama 3.1 8B · Inference | energy consumption | Hurt |
| Change PD disaggregation configuration (ablation) | Llama 3.1 8B · PD disaggregation workloads | energy per generation | No clear effect |
| Use larger batch sizes for Phi-3 Small with GQA rather than Phi-3 Mini with MHA (ablation) | Phi-3 Mini and Small · Chat on one NVIDIA A100 | energy per generation | Helped |
| Generate longer responses | Mistral Large compared with Mixtral 8×7B · LLM responses | energy consumption: +36% response length | Hurt |
Surprising findings
- Estimating energy from GPU TDP overestimates measured energy, by up to 4.1x for CodeGemma 2B on H100.
- Mistral Large (123B) per-request energy is comparable to diffusion models, and Phi-3 Small (7B) can consume less energy than the smaller Phi-3 Mini at large batch sizes due to GQA vs MHA KV cache size.
Limitations
- Benchmark is not exhaustive in models, tasks, or datasets, and becomes outdated; updates are batched due to monetary cost.
- Only flagship NVIDIA GPUs (A100, H100) are currently supported.
- No error bars: each configuration was run only once due to the high cost of benchmarking.
Experimental setup
Datasets: ShareGPT EvalPlus LLaVA instruction dataset PartiPrompts ShareGPT4Video GPQA
Benchmarks: ML.ENERGY Benchmark
Metrics: energy per request (J) time per output token (ms) average power draw (W) energy savings (%)
Baselines: Hugging Face LLM-Perf leaderboard MLPerf Power Hugging Face AI Energy Score TDP-based estimation
Builds on: Zeus vLLM iteration-level batching
Closest papers by meaning
- TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (AAAI 2026 · similarity 0.88)
- The Hidden Joules: Evaluating the Energy Consumption of Vision Backbones for Progress Towards More Efficient Model Inference (ICML 2025 · similarity 0.87)
- MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research (NeurIPS 2025 · similarity 0.85)
- BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices (NeurIPS 2024 · similarity 0.85)
- MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering (ICLR 2025 · similarity 0.85)
- Promoting Sustainable Web Agents: Benchmarking and Estimating Energy Consumption Through Empirical and Theoretical Analysis (AAAI 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.